Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

NYT: Healthcare.gov Project Chaos Due Partly To Unorthodox Database Choice

timothy posted about a year ago | from the even-if-it's-perfectly-nice dept.

Databases 334

First time accepted submitter conoviator writes "The NY Times has just published a piece providing more background on the healthcare.gov software project. One interesting aspect: 'Another sore point was the Medicare agency's decision to use database software, from a company called MarkLogic, that managed the data differently from systems by companies like IBM, Microsoft and Oracle. CGI officials argued that it would slow work because it was too unfamiliar. Government officials disagreed, and its configuration remains a serious problem.'" The story does not say that MarkLogic's software is bad in itself, only that the choice meant increased complexity on the project.

Sorry! There are no comments related to the filter you selected.

follow the money (5, Insightful)

ganjadude (952775) | about a year ago | (#45506927)

Who owns this company?

how much do they contribute to XXX???

There has got to be some reason that this DB that ive never even heard of (and i work with DBs, its not my main point of work but I know my way around DBs) got the gig over the more established players.

or, perhaps they went with it because it is less known and therefore reduce the risk of known attacks in other DB systems?

Re:follow the money (0)

danbuter (2019760) | about a year ago | (#45506957)

I'm guessing they were lowest bid.

Re:follow the money (5, Insightful)

Anonymous Coward | about a year ago | (#45506971)

from wikipedia,

The company’s main product, MarkLogic Server, is an XML based database management server.

This sentence has basically summed up the entire problem.

Re:follow the money (5, Funny)

Desler (1608317) | about a year ago | (#45507077)

They should have just used MongoDB. I hear it's web scale. *ducks*

Re:follow the money (2)

Charliemopps (1157495) | about a year ago | (#45507095)

WOW! No wonder they're having trouble. This has to be the worst idea for a database I've ever heard of. It definitely explains the problems they've been having.

Re: follow the money (3, Insightful)

sycodon (149926) | about a year ago | (#45507155)

XML soo sucks for db applications.

Re: follow the money (1)

EMN13 (11493) | about a year ago | (#45507297)

And you base this on what? Did the spagetti-monster tell you JSON was technically a better fit, or that XQuery doesn't work?

Re: follow the money (1)

Anonymous Coward | about a year ago | (#45507377)

Storing everything as XML means you're storing it all as strings (not very efficient). Now, if they're converting all the strings to binary data in order to store it, and then converting it back to XML to use it, then you're generating a lot of needless overhead because of the decision to store it as XML. Plus, searching XML sucks.

I'm not knocking XML as a means of handling data - it absolutely has it's place. But not in this size of a project that requires so much data and very high performance in order to handle the number of connections and requests.

Re: follow the money (1)

Pinky's Brain (1158667) | about a year ago | (#45507399)

I don't mind XML for stuff like Docbook ... but for databases what's the point? XML is an editing format with some down and upsides, using it as a storage format for data never edited in XML is just retarded.

At best it just provides a useless intermediary format which doesn't affect usability ... at worst you try to use XQuery to actually perform searches on large volumes of data.

Re: follow the money (4, Insightful)

Chris Mattern (191822) | about a year ago | (#45507445)

In fact, all of those would suck donkey balls. That is because XML, JSON and XQuery are all *data-interchange formats*. Using them to run your database internals is like using a screwdriver as a hammer.

Re:follow the money (0)

MightyMartian (840721) | about a year ago | (#45507163)

No fucking kidding. K-rust, I can only imagine how gawdawful slow such a database would be in every possible operation. Even if the XML itself is merely a wrapper for binary blobs, it's still an entire extra layer an engine is going to have to push through.

Why would anyone design such a product? Why would anyone buy such a product?

Re:follow the money (2, Interesting)

Anonymous Coward | about a year ago | (#45507207)

Because relational dbs are old and stuff. We don't tie onions to our belts anymore either, grandpa. NoSQL and XML are the future! A hipster I met at Starbucks told me so.

Re:follow the money (3, Funny)

Anonymous Coward | about a year ago | (#45507459)

Was this hipster drinking coffee or serving it?

Re:follow the money (2, Insightful)

WWWWolf (2428) | about a year ago | (#45507089)

Never attribute to malice that which can be explained by incompetence.

To me, this sounds like a pretty open and shut case of "Hey, I've heard that these 'NoSQL' database thingies are trendy these days. Let's use one of those!"

There's a difference between using fun, exciting new technologies and learning something new while doing that... and doing a project which stays in schedule and budget, is based on technology you already know thoroughly, and on which people's lives can depend (well, indirectly).

Re: follow the money (2, Insightful)

uradu (10768) | about a year ago | (#45507239)

You're exactly right! I've done tons of "exploratory" coding over the years myself, either using some new techniques or new products, just for the fun of it or to learn something new. But that would always be on small, low visibility projects where the consequences of potential poor performance or other issues would be insignificant. To tread new ground on something so big with national visibility is foolish. You'd want the most well established and known reliable and performant tools and techniques possible. I played around a bit with these object based databases in the early days, and for some uses such as simple dictionary type access to a serialized object they're good. But the query syntax is almost always XPath oriented and less than optimal for complex joins such as you would find in massive systems like this. I guess they've learned their lesson now.

Re: follow the money (1)

Anonymous Coward | about a year ago | (#45507303)

I guess they've learned their lesson now.

You give them entirely too much credit. In fact, they deserve none.

Re:follow the money (-1)

Anonymous Coward | about a year ago | (#45507465)

Never attribute to malice or incompetency that which can be explained by bribery.

FTFY

Re:follow the money (1)

Karl Cocknozzle (514413) | about a year ago | (#45507149)

There has got to be some reason that this DB that ive never even heard of (and i work with DBs, its not my main point of work but I know my way around DBs) got the gig over the more established players.

  or, perhaps they went with it because it is less known and therefore reduce the risk of known attacks in other DB systems?

Perhaps either is the case: Obscurity or donations... A third option is that it provides some feature that doesn't exist in Oracle/SQL Server/Cassandra. However in digging into their web-site it looks like it is some sort of wrapper/hybridized product that mates to Hadoop, which would make sense given the vast volume of data you're talking about managing for the Federal exchange, which I believe services 36 states.

But with that said, I've never heard of it either.

Re: follow the money (5, Insightful)

MyDixieWrecked (548719) | about a year ago | (#45507161)

I used marklogic when I worked at a previous job and after learning how it worked and understanding it better, it made our jobs incredibly easy. It just had a serious learning curve.

Marklogic is a nosql db, that uses XML for its object format and xquery for its query language. This thing is NOT mongodb. It actually works really well and allows for complex data modeling with the ability to do joins and have transactional isolation in making changes to the data as well as a really solid content processing framework with pipelining and all that jazz.

Now, I can't imagine a reason for using marklogic, or any non-relational db for a project like this. The only clue is that marklogic has a lot of government contracts; mostly for the military. So maybe that's why it was used. But the fact that they chose a database system that they weren't experts in for a project that had so much visibility speaks volumes on how mismanaged this whole project was.

Re: follow the money (1)

Mabhatter (126906) | about a year ago | (#45507235)

The only agencies that would need NoSQL type support are the ones spying on us. My take is that the insurance industry said they would only communicate in EDI-XML formats and somebody in management though that meant buying an XML-based system to be "compatible".

Re: follow the money (1)

MightyMartian (840721) | about a year ago | (#45507237)

This thing is NOT mongodb. It actually works really well and allows for complex data modeling with the ability to do joins and have transactional isolation in making changes to the data as well as a really solid content processing framework with pipelining and all that jazz.

So is SQL attached to any reasonably good content management framework.

Re: follow the money (4, Insightful)

Nerdfest (867930) | about a year ago | (#45507259)

To me, the fact that it was a major problem is an indicator of bigger problems. With systems like this, the database should be just a simple repository for persisting an retrieving domain models. If they want to do any non-trivial reporting on it the data should be replicated to a reporting repository. Treating the database a a simple persistence repository makes most database operations trivial, and better yet, decouples the database from the business, meaning that if you couldn't even manage the simple operations, replacing the layer is fairly trivial.

The problem is when people think the database is the starting point for a system, not just a tool to be used to support what your business logic is doing. It sounds like that's what happened here.

Re:follow the money (1)

jythie (914043) | about a year ago | (#45507169)

It is also possible that someone high on the chain was simply enthusiastic about it for no apparent reason. Several projects I have worked on used inappropriate databases because someone with enough technical knowledge thought it would be a good idea and their persistant insanity was enough to get it chosen.

Project Chaos (0)

Anonymous Coward | about a year ago | (#45506935)

Maybe they should choose better code names for their projects. This reminds me too much of Project Mayhem.

Noobs? (5, Insightful)

Anonymous Coward | about a year ago | (#45506941)

FTA: "An initial assessment identified more than 600 hardware and software defects — 'the longest list anybody had ever seen,' one person involved with the project said. "

Strikes me as none of these people seemed to have ever worked on large projects before.

Re:Noobs? (3, Funny)

Anonymous Coward | about a year ago | (#45507009)

You don't understand. That's not "600 bugs in the software" that they are talking about. "Thousands of bugs in software" is just one of 600 items that was a defect. Others included "Not enough servers", "Data connection to limited in capacity and speed", and apparently "No hablo espanol".

Re:Noobs? (3, Insightful)

larry bagina (561269) | about a year ago | (#45507447)

I counted 279 defects. (60 in the senate, 219 in the HoR). Had but 1 in the senate or 4 in the house been fixed, all this would have been averted. My kingdom for a horse, indeed.

Taxes... (0)

3seas (184403) | about a year ago | (#45506951)

I do not approve my taxes being spent for this.

Re:Taxes... (0)

Anonymous Coward | about a year ago | (#45507075)

I do not approve my taxes being spent for this.

I'm sure they took that into consideration...

Re:Taxes... (-1)

Anonymous Coward | about a year ago | (#45507123)

Too bad the election was decided by the 47% that pay no taxes.

Re:Taxes... (4, Insightful)

Bengie (1121981) | about a year ago | (#45507167)

The "47%" that people keep talking about are almost entirely retirees, adults going to full time college, or the disabled. The largest portion of the group are the retired people. Are you saying that people who have worked their whole like and are now retired, should not have a say in taxes?

Re:Taxes... (4, Informative)

Fwipp (1473271) | about a year ago | (#45507217)

Also the working poor - 47% pay no _income_ tax, but more than half of those people do pay payroll taxes, and even many those who don't (after deductions) still work.

They also pay sales tax, property tax, etc etc.

Re:Taxes... (1)

dugancent (2616577) | about a year ago | (#45507199)

Everyone pays taxes, income or not. Buy something? Sales tax. Pay utility bills? Taxed. Buy gas? Taxed.

Re:Taxes... (3, Informative)

Dereck1701 (1922824) | about a year ago | (#45507331)

Sorry to break it to you but that "statistic" is a bunch of BS. Everyone in the US pays taxes of some sort. I'm not particularly high on the tax scale (about $36k a year) and I estimate that i pay at least 30% of my income in local, state, federal and specific taxes (sales, gas, property, vehicle, etc). MAYBE some very poor (or very rich) people can avoid some of those taxes but EVERYONE pays at least a few of them.

Re:Taxes... (1)

Anonymous Coward | about a year ago | (#45507127)

That's ok, your share of taxes actually was specifically allocated to paying for a toilet in the Ozarks.

Re:Taxes... (1)

Anne Thwacks (531696) | about a year ago | (#45507349)

Please use MY tax for that - its way better than most of the stuff that tax gets used for! (On condition that it is a unisex toilet).

NIH syndrome (4, Insightful)

Gothmolly (148874) | about a year ago | (#45506953)

You could take a handful of proven DB technologies such as Oracle/DB/MSSQL, throw a web (Apache/IIS) and app (.Net/WAS/Jboss) front end to it, and it would work. Why did these guys fuck up the whole thing? It's like the scene in The Fountainhead when the second-rate architects smash up the plans and add their own stuff, "to express their own individuality". This could have been a solved problem - hell, it WAS a solved problem.

Re:NIH syndrome (-1)

Anonymous Coward | about a year ago | (#45506977)

NoSQL is a proven technology. It is used in production all over the place including where I work. Sorry about your late 80s db schemas.

Re:NIH syndrome (0)

Anonymous Coward | about a year ago | (#45507069)

NoSQL is an almost meaningless word. It is not a technology, it's a mass of different technologies. Some of which are complete crap.

Re:NIH syndrome (2)

MightyMartian (840721) | about a year ago | (#45507255)

Can someone explain to me how, for a project like setting up exchanges for Obamacare, NoSQL database systems is a rational choice? There are SQL-based systems like Oracle and MSSQL that can certainly handle recordsets of that size,and with that level of activity, and give you dialects of SQL sufficiently close to the norm that anybody with a reasonable level of RDBMS experience should be able work with it.

Re:NIH syndrome (0)

Anonymous Coward | about a year ago | (#45507417)

They probably have a bunch of XML feeds from different government agencies and private insurers, and some PHB saw "XML database" and decided it was perfect for the job.

Re:NIH syndrome (0)

Anonymous Coward | about a year ago | (#45506987)

but but but... NoGirlsAllowedSQL is WebScale!

WEBSCALE!

Re:NIH syndrome (0)

Anonymous Coward | about a year ago | (#45507049)

But it would not have worked if configured poorly, so I doubt using oracle, mssql or whatever would have helped, since they would have been quite likely to configure it just as poorly and to skip load testing anyways.

My university went for an Oracle solution(Software and hardware) for db for their intranet that handled everything and used it as a magic bullet since it cost so much. So first few months with the new system were a mess because it couldn't handle 100 simultaneous users due to combination of poor design that caused unnecessary amounts of lookups to be performed and bad configuration of the database, namely having a connection limit that was hit way before resources were at low. Of course the whole system could have been handled with postgre or even mysql, but they had a budget and burning it on license and hw seemed like a smart idea by the head of the IT(who was cursed by faculty of course and was cursed regardless of this particular decision).

Re:NIH syndrome (0, Flamebait)

Anonymous Coward | about a year ago | (#45507055)

Of course it was already solved. All the federal government really needed to do was allow health insurance to be sold across state lines so it could be portable, allow a range of plans to fit people's budgets with minimum coverage requirements, and pool high risk patients. Then, corporations or small businesses could have provided a tax free stipend to their employees who could purchase whatever coverage they wished.

Instead, we got a top down centralized planning crony corporatist version with a fucked up website and database, that requires young and healthy people to pay far more for insurance than required. You can't call it an unintended consequence, because it was well predicted that this administration would screw the pooch.

Re:NIH syndrome (5, Insightful)

Okian Warrior (537106) | about a year ago | (#45507065)

Why did these guys fuck up the whole thing?

All actions by Federal government players seem rational when viewed from their point of view.

The problem with most analysis, here or on other sites across the web, is in the assumptions. Specifically, if you assume that the government actions are for some benefit to the people, or efficiently completes some task (which is assumed to benefit the people), then nothing they do makes sense. That assumption makes for good outrage which can attract readers, but it doesn't answer the question of "why".

When government actions are viewed in light of more obvious motivations, everything it does makes sense. You start to see them not as an pack of incompetent, bureaucratic screw-ups, but as a self-sustaining, self-absorbed, engine of corruption.

I'm not aware of *anything* the federal government's done in the past two decades that's been in the interests of the people. All the useful stuff, the stuff you would want to keep across a reboot, was several decades ago.

I don't know the specifics of why an obscure database was chosen over the obvious players, but the reason wasn't "best utility for the people". Look for another reason.

Re:NIH syndrome (5, Insightful)

jythie (914043) | about a year ago | (#45507221)

Sad thing is, much of the behavior one sees out of federal contracts is due to taxpayer groups demanding anti-corruption measures. A great deal of the bureaucracy comes directly from people complaining about waste and demanding a complex audible process.

I would actually put forward that the problem is not the government, our government is pathetically weak in many ways. The problem are all the private groups flexing their muscles and making the government dance. Most of the people I have worked with on these kinds of projects are good people who just want to get stuff done, but they are weighted down by a nearly impossible set of requirements, many of which are mutually exclusive, dropped on them by outside groups who mainly are interested in showing off their power to their own people and have no investment in the project itself being successful.... and often have an active interest in the project failing since they can always use that as fuel for more personal power.

Re:NIH syndrome (0)

Anonymous Coward | about a year ago | (#45507079)

> It's like the scene in The Fountainhead
No, NO!!! That WORD, it BURNS, BURNSSS!!!

Auth problem - Oracle Identity Manager (2)

mveloso (325617) | about a year ago | (#45507415)

We were looking at what we could figure out about the architecture of healthcare.gov, and one problem is that it looks like it's using Oracle Identity Manager to manage the permissions of what users can/can't see. That means that OIM is burned in - and it's probably brutally slow, since every time you need to check a permission you go through OIM.

I'm not positive that's the case, but it fits given what pieces of the architecture I've seen. It would also explain why the system doesn't perform - permissions checking is always brutal, especially if you don't cache them. Caching permissions has more issues.

Stop with the excuses. (-1)

Anonymous Coward | about a year ago | (#45506959)

It's a clusterfuck. And it was designed from the start to be a clusterfuck because half the people involved wanted it to fail. So they could blame the other half.

Re:Stop with the excuses. (2, Informative)

Anonymous Coward | about a year ago | (#45507019)

It was designed from the start to pave the way for single payer health care.

Re:Stop with the excuses. (5, Insightful)

DarkOx (621550) | about a year ago | (#45507053)

Bullshit. Since the law has been passed almost the entire implementation has been up to the administration. Which in case you had not noticed is lead by partisans in favor of the affordable care act.

If the GOP can be accused of anything material in terms of interfering with the implementation it would preventing the law from being amended, and if it needs amending there is your proof it was bad law to begin with. Sorry Obama owns this and really HE owns it not even the larger DNC as its HIS branch of government that has been responsible for implementation from the moment HE signed the thing.

*IF* it fails its either because it was unworkable in the first place as written and should not have been enacted or because the Obama administration failed to execute. Those are really the only honest high level conclusions.

The argument well because they had to pass the Senate bill as is it did not get the usual fixes and tweaking is why its got so many problems is also bogus. They had to pass the Senate bill by abusing the budget reconciliation process to deny the minority party in the House at the time its rights; they knew if usual procedures were followed it would never have become law. So again it either should not have been enacted or the administration has failed to execute.

Re:Stop with the excuses. (1)

Assmasher (456699) | about a year ago | (#45507099)

Since the law has been passed almost the entire implementation has been up to the administration.

The implementation is, for the most part up to the administration and they certainly have screwed the pooch on this; however, the GOP can quite easily be accused of willfully sabotaging the process through the ACA lawsuits and intentional state exchange shortcomings (not to mention a myriad of other issues that relate to the shutdown, refusing ancillary funding out of spite, and issues with the Supreme Court.)

In any case, even if the GOP are being total jerks about the law, healthcare.gov requires a software solution that has been patterned thousands of times. It's a total fumble by the Democrats.

The party system should simply be banned outright.

Re:Stop with the excuses. (1)

DarkOx (621550) | about a year ago | (#45507325)

GOP can quite easily be accused of willfully sabotaging the process through the ACA lawsuits

Attempting to sabotage it maybe, succeeding no. There was never even so much as a stay issued, as far as I am aware, preventing the administration from getting started on implementation. If I am wrong on that please correct me but I don't think the court cases could have had much impact on the implementation.

The medicare expansion issue is going to create a coverage gap, in the states that did not do it. That is about the only *problem* the legal challenges successfully created.

As far as states not implementing their own exchanges the law always intended to provide a federal solution for them; so that isn't much of an excuse for its failure.

Re:Stop with the excuses. (1)

I'm New Around Here (1154723) | about a year ago | (#45507105)

To be fair, the AC didn't specify Republicans and Democrats. It could have been internal groups behind the website itself that wanted to blame other internal groups, to pump up their own importance. That could be what he meant.

.
But, yeah, he almost certainly made a laughable dig at the GOP there.

MarkLogic = NoSQL (4, Interesting)

NeverWorker1 (1686452) | about a year ago | (#45506979)

A little googling turns up that MarkLogic's offering is NoSQL. Without getting into the whole SQL/NoSQL debate, I can't help but noting that this is clearly relational data with a fairly limited number of records (clearly there can't be more than 300M people listed) and for which ACID is (or should be) a major concern.

Re:MarkLogic = NoSQL (1)

mobby_6kl (668092) | about a year ago | (#45507035)

But how would a traditional relational database scale to the 1 billion, or 1,000 billion users, huh? Did you think about the need to future-proof the application?

Re:MarkLogic = NoSQL (2, Funny)

Anonymous Coward | about a year ago | (#45507061)

They should have used MongoDB, because MongoDB is webscale.

Re:MarkLogic = NoSQL (0)

Anonymous Coward | about a year ago | (#45507073)

Let me know when that hits 1 billion.

http://www.census.gov/popclock/

Re:MarkLogic = NoSQL (-1, Offtopic)

I'm New Around Here (1154723) | about a year ago | (#45507145)

About two years after they open the borders to all illegal immigrants who promise to vote Democrat.

Re:MarkLogic = NoSQL (0)

Anonymous Coward | about a year ago | (#45507091)

Mod: I think this should be marked as humor

Re:MarkLogic = NoSQL (1)

Assmasher (456699) | about a year ago | (#45507125)

AKF cube.

Healthcare.gov isn't a particularly difficult scaling problem to solve. You can 'swim lane' the hell out of it.

It's not like it's a social network where people need to be constantly sharing things and an arbitrarily large (potentially huge) group of people need to be notified of everything each of them does/doesn't do...

Re:MarkLogic = NoSQL (1)

DarkOx (621550) | about a year ago | (#45507137)

But how would a traditional relational database scale to the 1 billion, or 1,000 billion users, huh? Did you think about the need to future-proof the application?

I don't know if you are kidding or not but I'll try anyway.

The US population might hit 1 billion inside the period where something form of the ACA is intact in a recognizable way. Unlikely but its possible. Using partitioning schemes it should be fairly simple to scale on traditional database technologies because for most queries you don't need to know about Abby to handle a transaction regarding Bob. There are some where you do but even those are mostly simple 1:1 update operations, you might need to set Abby's spouse attribute to Bob's SS number or something; never the less there are lots natural ways to partition the data which should make scaling out across more RDBMS servers workable; at least out to the 1 billion mark.

At 1,000 billion we will probably be so concerned with were our next glass of water is coming from that who has health insurance and who is claiming to be President for that matter won't really matter to anyone.

Re:MarkLogic = NoSQL (1)

swilver (617741) | about a year ago | (#45507229)

It would scale just fine, assuming you make proper indexes for your queries.

Since the general use case is probably gonna be something like "pull up records for patient that is sitting infront of me and edit them", I think an index lookup + atmost a couple of dozen records pulled from related tables will be handled just fine.

Any other use cases that involve statistical analysis on all data can be handled by copying it to specialized DB's every few weeks.

Re:MarkLogic = NoSQL (3, Informative)

gmuslera (3436) | about a year ago | (#45507323)

Google and Facebook seem to be doing pretty well having that order of users. Facebook even uses simple mysql replication and their pool scanner [facebook.com] . And health system is not a social network, there is no so much connection between different users.

Re:MarkLogic = NoSQL (5, Interesting)

Anonymous Coward | about a year ago | (#45507045)

A customer at work has a MarkLogic database. I don't know what version it is, but I assure you IT IS HORRIBLE. It's like... an XML database or some shit. Just awful, and extremely confusing to use.

A couple years ago I was asked to do automatic database backups. The only documentation I could find for backing up the database requires logging into a web interface I had no idea existed on an obscure port and clicking a backup button. I literally had to write a perl script to fake a browser and do this just so they could get a database dump.

Do not use this product. Please.

Re: MarkLogic = NoSQL (4, Interesting)

MyDixieWrecked (548719) | about a year ago | (#45507215)

Marklogic, afaik, is the only acid compliant nosql solution that exists.

Re: MarkLogic = NoSQL (2)

xaxa (988988) | about a year ago | (#45507443)

Marklogic, afaik, is the only acid compliant nosql solution that exists.

Neo4j is an ACID-compliant graph database (so it's NoSQL, but that word is a bit pointless).

It's also the most useful non-relational DBMS: relationships become first-order entities, having a type and properties.

http://www.neo4j.org/ [neo4j.org]

New tech can be great, but tough to find skills... (0)

Anonymous Coward | about a year ago | (#45506989)

This is the problem every time you try to use some nifty new tech that hasn't matured yet. Heaven forbid your skills lead decides to leave for greener pastures (and believe me, there are a lot of greener IT pastures than government contracting...), you're stuck trying to replace him or her and you soon find out that there's only 10 people in the entire universe that actually are proficient with WizBangDB.

Re:New tech can be great, but tough to find skills (0)

Anonymous Coward | about a year ago | (#45507041)

her and you soon find out that there's only 10 people in the entire universe that actually are proficient with WizBangDB.

And one of those 10 is in the Tardis, asking the Doctor what went wrong with healthcare.gov

Blow to NoSQL movement (4, Insightful)

ilsaloving (1534307) | about a year ago | (#45507007)

And here is a fantastic example of what happens when hype trumps common sense. NoSQL is the new hawtness, and apparently the dumbasses running the project wanted to be part of that. Now MarkLogic, and NoSQL in general, will have a massive blow to their reputations, and it's unknown how badly this will hurt them.

As someone who has done databases for a long time, I have very little respect for NoSQL, but that is mostly because everyone keeps trumpeting it as a killer of traditional databases. There are scenarios where NoSQL systems are an ideal fit. However, NONE of those scenarios require data to be very reliably stored in a guaranteed and predictable way.

If you don't get your tweets or your friends facebook posts as soon as they are posted, no one will really care. But for something as truly important as health insurance coverage? Are you f__king kidding me? And that's just from a reliability standpoint. Nevermind the fact that NoSQL is currently at the wild west stage where nobody is compatible with anybody else, there is nothing resembling a standard set of APIs between products, making it very difficult to develop expertise.

Sounds like the Gov was just begging for problems.

Re:Blow to NoSQL movement (0)

Anonymous Coward | about a year ago | (#45507087)

Which makes me wonder what bugs exists in that system that aren't very apparent. I mean what data loss issues are there that don't just jump out like something that's outright broken and will never work.

Re:Blow to NoSQL movement (1)

ducomputergeek (595742) | about a year ago | (#45507093)

Just had this debate with a current project with some wanting to use a NoSQL solution for the whole thing. Problem is most of the data is relational and I stuck my foot down and said we're using PostgreSQL for anything that needs to be retained. That mean users accounts & transaction records, and really all the data is relational.

Now there are other elements, like the chat system and distributing JSON strings to large numbers of persistent clients that seem a perfect fit for a NoSQL database. Since the JSON strings are basically information caches from the backend database to be widely distributed so what if the NoSQL db crashes. Spin up a new instance and reload the data from the main DB and start distributing again. Chat messages only need to persist for a few minutes at most. So honestly a crash or glitch and frankly very little of value would be lost.

Re:Blow to NoSQL movement (2, Interesting)

Anonymous Coward | about a year ago | (#45507139)

The agency imposing their "unconventional" choice of DB, and this sentence:

"They ordered CGI technicians to drive from their offices near Dulles International Airport in Virginia to the agency headquarters near Baltimore to review their code with government supervisors."

sum up everything that went wrong.

Basically you had a bunch of arrogant idiots who thought themselves experts (which they may well be, _in the healthcare domain_) who not only ignored the views of the _software/IT_ experts ("technician" my a**, senior software engineer, more like), who they saw as plebes, but proceeded to question their work.

When will those guys realize that software developers are not blue collar workers?

That they went for the usual bigco contractors and their hordes of yes-men certainly didn't help.

Re: Blow to NoSQL movement (0)

Anonymous Coward | about a year ago | (#45507371)

Politicians (like lawyers) areexperts! They just read a few articles in a few hour period and *bam*, they're instant "experts" on the subject.

Re:Blow to NoSQL movement (2, Insightful)

Anonymous Coward | about a year ago | (#45507141)

NoSQL predates SQL because we were storing stuff back then to. Relational databases are nice for consistency, but you sacrifice a lot of control and speed improvements for that consistency. A good NoSQL database and a good NoSQL programmer will always outperform SQL based applications. However, it means you need to have programmers capable of operating at that level AND you need to have solid strongly enforced conventions.

I also think that you are largely misreading the problem. Being able to get a quote and select your insurance isn't a strongly time sensitive endeavor. Really, most of the information you need is static. And as soon as you put data into a properly configured NoSQL database it should be available everywhere. There really shouldn't be race conditions of import for this sort of system. The system essentially asks you a few questions. You give it those answers and it returns a set from which you pick. After agreeing to some conditions you submit and are done; at least, that seems like the best thing.

I honestly think either platform, configured correctly, would work. And I program in a NoSQL database that predates C. Not C++, Not C#. I mean C. This kind of system scales just fine when you build indices and if your NoSQL database presents the right tools. I don't know anything about MarkLogic; but would you judge C by how a programmer operates in whitespace?

If you think NoSQL can't or doesn't perform in high risk environments, think again:
http://robtweed.wordpress.com/2012/10/21/mumps-the-universal-nosql-database/

Re:Blow to NoSQL movement (1)

Anonymous Coward | about a year ago | (#45507147)

MarkLogic's NoSQL managed to land a giant lucrative contract for the venture capitalists backing the operation. Failure to provide an actual working solution really doesn't matter to the profiteers. In that sense, MarkLogic has been a massive success. See also, Microsoft Corporation, and decades of foisting off third-rate software (with massive known security and reliability failings) for supposedly "mission critical" business and government applications, raking in billions and billions in profit. If the corporate PR spinmeisters do their jobs right, public blame for system failure can be placed directly on President Obama's shoulders, and MarkLogic's products will be the "big data solution of choice trusted for national-scale deployment."

Re:Blow to NoSQL movement (1, Insightful)

smpoole7 (1467717) | about a year ago | (#45507301)

> MarkLogic's NoSQL managed to land a giant lucrative contract for the venture capitalists ...

+24 insightful and informative. This.

Here's another angle on it. Haven't you ever wondered why everything associated with the government takes longer and costs more?

I once worked with a guy who did government contracts all the time. He said, "you deliberately underbid* to make sure you get the deal. Just be sure to put a clause in there protecting yourself when the government (which is Rule By Committee, remember) makes changes. After you get the contract, you KNOW the government will make changes. You can charge whatever you want and take as long as you want."

* he also said, and I quote: "and you buy the people making the decision a bunch of hookers and get them drunk." Basically his exact words. You'd probably have to update that now to a government that includes many females in management positions, but you get the idea. :)

Re:Blow to NoSQL movement (3)

Anne Thwacks (531696) | about a year ago | (#45507393)

I once worked with a major multinational who did government contracts all the time. They said the exact same things (but you failed to mention the brown envelopes filled with banknotes).

Re:Blow to NoSQL movement (3, Insightful)

anon mouse-cow-aard (443646) | about a year ago | (#45507157)

Why do people insist that NoSQL means losing data and inconsistency? What you are losing with NoSQL real-time consistency horizontally across large number of instances spread over multiple data centres. When you are doing a transaction, and it should be "eventually consistent." meaning on the order of minutes. So if someone, somewhere else, who you do not know about and are not interacting with asks about your data, it might be a few minutes old. ACID makes it so that random person will get an upto the milli-second accurate answer. That makes transactions orders of magnitude slower, and much more complicated to scale. When you relax that constraint and let the people doing the transactions (the person, and the providers they are dealing with) get the right answer immediately, but just post the transaction so that the backups etc... get informed in due course, you gain much simpler scaling. The choice of NoSQL vs. SQL is not about the importance of the transaction. It is about application scaling and design, and the biases (tastes?) of the team doing the work.

Re:Blow to NoSQL movement (4, Interesting)

Assmasher (456699) | about a year ago | (#45507251)

I ran into this recently at a company whose new head of engineering was talking to me (an outsider) about the technology problem they had to solve and I thought it sounded very traditional and simple except they'd need to carefully plan for horizontal scaling.

Basically a potentially huge number of devices (in the range of millions) would be reporting in periodic data that had to be stored and potentially evaluated in real-time. The data was quite easily swim laned by geolocation and the data had no appreciable inter-related significance. So basically, one piece of data from one device had nothing to do with any other device's information except in the general sense that can come from a more heuristic correlation of their data.

I should mention that the new engineering head and I had already (together) handle a situation very similar to this at a previously successful software company.

Well, the new engineering head had inherited an external architect who had different ideas. All of these different ideas involved things like Cassandra over Hadoop, AMP/Spark, BDAS. He showed me a diagram of the technologies he wanted to integrate and I'd never heard of almost half of them (and I deal with scaling issues all the time), this diagram had about 15 different technologies stacked together. It was crazy - all to solve a relatively simple data volume problem.

Almost needless to say, I advised otherwise, and afaik they're going the bid data way because it will make it easier for them to pull in VC money since shockingly few VC's actually evaluate technology before they put money in (I do this for VCs also, and other VCs wonder how I get paid to do this, lol.)

MarkLogic is an XML repository, not a RDBMS (5, Interesting)

CyberLeader (106732) | about a year ago | (#45507013)

"Some people, when confronted with a problem, think 'I know, I'll use XML.' Now they have two problems."

-JWZ

MarkLogic is an XML database, not a relational database, so if your data primarily consists of XML content then it's the right tool for the job. Sounds like the vendor building the system had a favorite hammer and decided that a rather traditional database problem looked like a nail.

MarkLogic itself is fine if your data fits neatly into an XML schema, but with healthcare.gov that tree is probably enormous and hard to optimize for DB activity.

Executive Fiat will fix this!!!! (-1, Flamebait)

Anonymous Coward | about a year ago | (#45507017)

Have no fear, citizens! All that glorious leader Obama needs to do is declare that the website must get fixed by the end of November and by golly, it WILL get fixed by the end of November. ...And then he'll amend it to say "by the end of December -- PERIOD!" ...And then he'll later say months later when it is still in shambles "In all fairness I didn't say December 2013".

Wonder which Reps own share on MarkLogic (0)

Anonymous Coward | about a year ago | (#45507029)

Would make for interesting reading

How cait it be the problem (1, Insightful)

arthurpaliden (939626) | about a year ago | (#45507057)

Well they did have the flashiest ads which used lots of buzzwords.

Re:How cait it be the problem (0)

Anonymous Coward | about a year ago | (#45507233)

It could've been worse. Word is they considered HarryLogic and DickSolutions before settling on MarkLogic.

Govt contracts (0)

Anonymous Coward | about a year ago | (#45507063)

Anyone who has worked for anything in the govt knows most contracts are a formality. The deal is done before the job is pulbished seeking bids.

Open Source (2)

Murdoch5 (1563847) | about a year ago | (#45507067)

Why not just use MySQL, MongoDB or MariaDB? At least use a database system that has good support, an easy learning curve and loads of followers. That beings said, proper testing would easily of mitigated this entire issue.

Re:Open Source (4, Insightful)

NeverWorker1 (1686452) | about a year ago | (#45507083)

Because Postgres exists.

Re:Open Source (0)

Anonymous Coward | about a year ago | (#45507193)

i am mostly surprised they didn't use oracle. because that is the fastest way to waste money.

healthcare.gov (0)

Anonymous Coward | about a year ago | (#45507107)

I can create an account, but I can't fill out the application form. The website gives me an error message when I click the button to go to the next page. arg

CGI = Computer-generated imagery

Impressive (1)

SuperCharlie (1068072) | about a year ago | (#45507135)

Just when I think.. wow.. you couldnt screw this up any worse... they step up and prove me wrong.

What happened to Vivek Kundra (0)

Anonymous Coward | about a year ago | (#45507153)

The guy who Obama appointed the first US CTO with great fanfare? He was supposed to be heavily involved in getting stuff like this right.

Oh right, he resigned in 2011 to pursue a job in the private sector. I don't think that absolves him responsibility. He should've stayed to get the health care rollout right, then he could leave.

Unorthodox (1)

Horshu (2754893) | about a year ago | (#45507177)

I've lost count of how many projects I've been on where the architect decides to "make his mark" by using unconventional design choices. Then the project gets stuck in a dev hell where the actual developers struggle with either integration headaches or difficulties with the code not acting like they expect. There's something to be said for plain vanilla.

Re:Unorthodox (0)

Anonymous Coward | about a year ago | (#45507257)

In an data-driven organization, DBAs rule. Sorry.

US Federal Government Systems Projects (5, Interesting)

vrhino (2987119) | about a year ago | (#45507231)

I worked as a contractor developing a system at FDA. It lasted for 5 years. Inside the Beltway, it's pretty much the same all over. Dysfunctional communications and ridiculous breakdown of authority not corresponding to lines of management. No accountability. Project management requirements that have never been followed by any project. No commitment to the output of requirements gathering. No requirements change control. No performance engineering. Inadequate testing. No acceptance process by the government. IT groups with oversight for contractor output that have never written a line of code. All in all, pretty sick and ugly. Prior to my project there were 5 failed attempts. My project followed PMI practices, worked them hard and succeeded.

MarkLogic's Pitch (1)

conoviator (1991610) | about a year ago | (#45507265)

From a slide that promotes MarkLogic's appropriateness for the health exchange's particular set of challenges:
  • - Highly complex data in many formats that change often and have varying quality
  • - Massive amounts of data at high velocity; highly transactional
  • - Highly structured data, but multiple and changing schemas

See: http://assets1.csc.com/innovation/downloads/LEFBriefing_MarkLogic_031512.pdf [csc.com] (slide 23)

My two cents:

  • When faced with a very complicated software project, use what's been proved to work.
  • Why would the CMS (Centers for Medicare & Medicaid Services) dictate this particular less common technology? Very strange.

Unnecessary complexity is actually bad in itself (2)

JoeyRox (2711699) | about a year ago | (#45507339)

From the summary:

"The story does not say that MarkLogic's software is bad in itself, only that the choice meant increased complexity on the project."

Unless that complexity was necessary to solve a problem, then it is in fact bad.
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?