Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×

Data Sharing, Government Style 96

Posted by timothy from the popular-and-fun-just-like-ada dept.
rowama writes "The Department of Homeland Security and the Justice Department have been collaborating to develop an XML-based model for data sharing. After less than a year since the initial release, in October 2005, the National Information Exchange Model (NIEM) 1.0 Beta is out. It's big, really big. There are no less than 9 namespaces and plans for future expansion. Contact your local government contractor, with resume in hand, and you may be one of the lucky developers to implement NIEM-capable software."
This discussion has been archived. No new comments can be posted.

Data Sharing, Government Style More

Data Sharing, Government Style

Comments Filter:

  • Bonus advantage (Score:5, Funny)

    by nizo ( 81281 ) * on Monday July 17, 2006 @05:33PM (#15733923) Homepage Journal
    Contact your local government contractor, with resume in hand, and you may be one of the lucky developers to implement NIEM-capable software.


    As an added bonus you can add a wee bit of code to make sure your name never ends up in these databases.

    • Hey, don't laugh. This could be bigger than Ada.

    • Re:Bonus advantage (Score:5, Interesting)

      by Lally Singh ( 3427 ) on Monday July 17, 2006 @05:41PM (#15733998) Journal
      There are some large ethical questions programmers have to ask themselves when taking on jobs these days. After my last DoD gig, I've really started filtering what opportunities I'll consider. Mass surveillance, for example, is something most of my contacts know I won't touch.

      As for the 9 namespaces, it's actually pretty reasonable. From TFA:
      xmlns:u="http://niem.gov/niem/universal/1.0"
      xmlns:s="http://niem.gov/niem/structures/1.0"
      xmlns:c="http://niem.gov/niem/common/1.0"
      xmlns:j="http://niem.gov/niem/domains/justice/1.0"
      xmlns:emer="http://niem.gov/niem/domains/emergency -management/1.0"
      xmlns:im="http://niem.gov/niem/domains/immigration /1.0"
      xmlns:ip="http://niem.gov/niem/domains/infrastruct ureProtection/1.0"
      xmlns:int="http://niem.gov/niem/domains/intelligen ce/1.0"
      xmlns:it="http://niem.gov/niem/domains/internation alTrade/1.0"

      • Re:Bonus advantage (Score:5, Funny)

        by Kesch ( 943326 ) on Monday July 17, 2006 @05:47PM (#15734038)
        They need more terrorist references. I also see a distinct lack of Thinking of the Children(TM). Also, the namepsace count just isn't bloated enough. I don't believe this spec is up to government work yet.

      • Re:Bonus advantage (Score:4, Funny)

        by Quiet_Desperation ( 858215 ) on Monday July 17, 2006 @06:10PM (#15734170)

        After my last DoD gig, I've really started filtering what opportunities I'll consider.

        I also go looking for the projects that have the potential to kill the most people, but then again I'm an utter misanthrope. :D

        Although I have to admit that cybernetic, remote controlled stealth shark thing DARPA announced a while back had my interest. No killing the enemy, but it's fricken stealth sharks, man! You know I'd fight for comm lasers to burst the data back to base.

        Don't worry, I'll take those mass surveillance jobs. I'll do them really well, too. Sleep tight. :)

        • I know you're joking (well, maybe), but I often think that the Slashdot crowd fails to appreciate how many people there are in the world -- very smart people, in fact, in many cases -- who are more than willing to take the "dirty" jobs.

          Particularly if they're interesting dirty jobs.

          The fact that what you're doing can be used to kill people fades away into relative unimportance pretty quickly, if there's a cool technical challenge to be solved, and the salary is right, and the people you get to work with are

      • Re:Bonus advantage (Score:1, Funny)

        by Anonymous Coward
        They missed some of the classified ones:

        xmlns:wc="http://niem.gov/niem/domains/warCrimes/1 .0"
        xmlns:t="http://niem.gov/niem/domains/torture/1.0"
        xmlns:uca="http://niem.gov/niem/domains/unConstitu tionalActivities/1.0"

        or do they just fall into the 'common' namespace?

      • Re:Bonus advantage (Score:3, Insightful)

        by Soong ( 7225 )
        under which namespace will I find the <pork/> tag?

    • Or better yet, think of this project in the same way you'd think of implementing an Obfuscated C Code contest entry - how horrible can you make the code, and still get paid. Make it as painful as possible for Big Brother to go data mining.

      That, or use some ethics and don't take work like this.

  • thank Government for databases (Score:5, Insightful)

    by yagu ( 721525 ) * <{yayagu} {at} {gmail.com}> on Monday July 17, 2006 @05:34PM (#15733935) Journal

    Meanwhile grandma is still taking off her shoes and getting wanded at the airport. Nice to know yet another debacle is launched. Here's hoping they're as successful as they have been with the new Air Traffic Control System.

    • Meanwhile grandma is still taking off her shoes and getting wanded at the airport.

      As objectionable as this is, I think the bigger problem is the racial scanning that goes on at these airports. There are large groups of Middle Eastern people living in the US. Have they attempted any massive terrorist operation? To grandma, it's just an inconvenience. To these people, this is prejudice. Why do people go crazy over some dumb psp ad (which didn't even make it to the US) and skip over these issues?

      • which didn't even make it to the US

        Yeah, well neither do the people [wsj.com] who've been racially profiled onto the no-fly-list once they've left. (registration free link [informatio...house.info])

      • Re:thank Government for databases (Score:2, Interesting)

        by Anonymous Coward

        As objectionable as this is, I think the bigger problem is the racial scanning that goes on at these airports.

        No, the bigger problem is making government agents into robots. They only follow procedures and aren't allowed to think for themselves (or heaven forbid, take initiative), for fear that someone could say that they were performing racial profiling. We are more afraid of the political repercussions of a few racial discrimination cases than the repercussions of planes being bombed or flown into major

    • Re:thank Government for databases (Score:4, Insightful)

      by Anonymous Coward on Monday July 17, 2006 @06:32PM (#15734275)
      Meanwhile grandma is still taking off her shoes and getting wanded at the airport.

      The knowledge that she will get the same treatment as the rest of us is the one thing keeping grandma from demanding body cavity searches for the rest of us.

      I, for one, am glad that we don't live in a world where grandma gets waved through security with a smug little smile on her face while I get directed to the body cavity search room to take it in the rear to appease grandma's paranoid fears of "all those terrorists".

  • Aliens? (Score:4, Funny)

    by SatanicPuppy ( 611928 ) * <SatanicpuppyNO@SPAMgmail.com> on Monday July 17, 2006 @05:35PM (#15733943) Journal
    Okay...I can see the need for u:SuperType->u:ActivityType->c:ActivityType->im:Al ienEncounterType...I mean, we're bound to encounter aliens at some point, right?

    But im:AlienStudentDisciplinaryActionType? Planning for Alien encounters is one thing, but planning for dealing with them in our school systems seems like bureaucratic bloat to me. I don't think the Red Staters will be down with their taxes going to teach godless little green people.

    (end humor tags)

    • Re:Aliens? (Score:4, Funny)

      by __aaclcg7560 ( 824291 ) on Monday July 17, 2006 @05:40PM (#15733984)
      You got to account for AlienIllegal, AlienET and AlienOfTheWeirdAndPissedOffVariety to cover all your bases. Assuming that some AlienET doesn't already own all your bases.
    • I don't think the Red Staters will be down with their taxes going to teach godless little green people.

      Godless??? How do you think the Red Staters will respond to learning that the little green people, arriving in their brand new model 6006 JHVH craft, seeded all life on Earth - Thus making them our gods?

      Enki forbid that their advanced civilization might have very neatly solved the whole abortion issue by promoting homosexual activity as a form of birth control... ;-)

      I can hear their cute little hea
    • I think your real question should have been.. how does alt.alien.vampire.flonk.flonk.flonk map into these namespaces?
    • As an hsitorian and far less of a programmer, I must say you penetrate both worlds with the same vector clear in sight. Dry wit always most appreciated too, whoever you are my compliments. Nice to see others in numerous locales awakening to the "bloat" to which I must too often refer, historically most frequently labeled "government."

  • Use XML. (Score:1, Funny)

    by Anonymous Coward
    The solution to every problem begging for more markup and extra layers of abstraction.

  • TSDB (Score:2, Informative)

    by 9x320 ( 987156 )
    Neato. Maybe now they'll make less errors in that Terrorist Screening Database they have. You know, the one that has the names of over 250,000 people tagged as terrorists used in everything from no-fly lists to border crossings ever since the administration wanted all such watchlists to be consolidated into a single big one. That one the NSA probably uses. That one that, according to Department of Justice Inspector General reports [iphide.com], may be riddled with errors.

    Read the Department of Justice and Department of

    • Re:TSDB (Score:3, Informative)

      by 9x320 ( 987156 )
      3,673 records had been removed from the Terrorist Screening Database since its creation in June 2004 until this DOJ Inspector General report came out in May 2005. The page of the Inspector General report clarifies that when a possible misidentification of a suspect with Terrorist Screening Database records is found by the Terrorist Screening Center, the Quality Assurance team reviews the information with the agency (either the National Counterterrorism Center or a certain FBI unit) that nominated the record
    • Now, according to an August 2004 redacted DHS Inspector General report [backfox.com], the Terrorist Screening Center has a phone number and e-mail address where "federal and local organizations," presumably by which they mean law enforcement, can call about mistakes in the Database, suggesting there has been mistakes in this Database. A Washington Post story says an anonymous official said a 'very, very small fraction' of the names in this database are U.S. citizens.

      If these names are used by the NSA in deciding whose ph

    • How can less errors help with a method that was completely flawed from the start? I mean, didn't it ever occur to anyone in DHS that normal, law-abiding American citizens might have names similar to at least some of those that are on "the list," and that because of this, they'd be subject to baseless abuse by those relying on it? After all, someone would never attempt to identify themselves as someone other than who they really are...nah, that would never happen.
  • National Information Exchange Model

    On some level, information has always been exchanged between these powers. Now they're using XML. Cool.

    • Re:bah (Score:3, Funny)

      by deathy_epl+ccs ( 896747 )

      On some level, information has always been exchanged between these powers. Now they're using XML. Cool.

      Yes, but see... with the advent of XML, that information exchange is now more than just "Uh-uh, not gonna tell ya!"

      Now, they have a name-space that includes the ability to tack a "NYAH NYAH" on to the end of the statement.

    • Well, NIEM does incorporate and glom together some existing XML "standards" - Global Justice XML, for example. But there's nobody in gov't who's really up to speed with that either.

      I think you'd be shocked at how little information really is being exchanged currently.
  • The Senate is going to convene hearings on why Microsoft's new Office Open XML format shouldn't be used instead.

    Of was that a different Senate?

  • The National Information Exchange Model (NIEM) 1.0 Beta is out. It's big, really big.

    But that's peanuts to space.

    • Re:How big? (Score:5, Funny)

      by Tackhead ( 54550 ) on Monday July 17, 2006 @05:50PM (#15734060)
      > > The National Information Exchange Model (NIEM) 1.0 Beta is out. It's big, really big.
      >
      > But that's peanuts to space.

      I mean you think there's a long list of entities in the markup for your CSS/AJAX/Web2.0 project's folksonomy, but that's just peanuts to the NIEM," and so on.

      After a while, the spec settles down a bit and tells you things you really want to know, like the fact that the fabulously corrupt city of Washington D.C. is now so enamored of the cumulative fiscal erosion by ten billion visiting lobbyists a year that any net imbalance between the amount you donate and the amount you receive in federal contracts whilst on the take is surgically removed from your bank account when you leave: so every time you go to K Street, it is vitally important to get a receipt... and falsify it.

  • Obvious bloat. (Score:5, Insightful)

    by SatanicPuppy ( 611928 ) * <SatanicpuppyNO@SPAMgmail.com> on Monday July 17, 2006 @05:49PM (#15734050) Journal
    Just glancing at it, I can see problems. XML is too often used for databases when it shouldn't be, but there are similarities, and just looking at it I can see that it violates one of the most basic database design principles: normalization [wikipedia.org]

    Just as an example, there are three different namespaces dedicated to the various FIPS (Federal Information Processing Standards)...To three different STANDARDS.

    I'm no expert on government info, and I just looked at this thing for the first time, so maybe it's brilliant and I'm ust not seeing it, but it sure looks a lot like they've fallen victim to a database noob mistake, and created a monster tree with disproportionate crazy branches everywhere, and that is bound to cause relational problems, redundant data, and warped design challenges.

    • Re:Obvious bloat. (Score:4, Insightful)

      by punkinabox ( 922181 ) on Monday July 17, 2006 @06:01PM (#15734122)
      Well, they said data sharing, not data storage.

      • Re:Obvious bloat. (Score:4, Insightful)

        by truthsearch ( 249536 ) on Monday July 17, 2006 @06:04PM (#15734137) Homepage Journal
        Exactly. We're talking about data interchange between systems, not single system efficiency.
        • Normalization is still desirable, just because of its efficiency. No wasted space.

          Just looking at the tree representation of the class structure, I don't know what they're thinking...Tier 1, the "Supertype" level, has two nodes (not counting the units-of-measure bits). Okay... Tier 2 has more than a hundred! More than all the Tier 3s combined! That's not moving from simple to complex, which is what the goal of structured data should be! It's just throwing stuff in a pile, and telling people what the pile lo
          • If your goal is to create yet still more bloated systems with yet still more arcane data constructs, this is a good start.

            Well, it's proved to be a pretty good business model so far ... why mess with a good thing?

    • Re:Obvious bloat. (Score:2, Funny)

      by kfg ( 145172 ) *
      that is bound to cause relational problems, redundant data, and warped design challenges.

      Do me a favor, don't tell them.

      KFG
    • The NIEM is for data exchanges only. It is explicitly not intended as a guide for structuring your internal data storage.

      Support for multiple standards like FIPS tables is intentional. (No one wants to try and fiat one.) A mechanism for marking one as preferred is in the works.
    • Just as an example, there are three different namespaces dedicated to the various FIPS (Federal Information Processing Standards)...To three different STANDARDS.

      I'm no expert on government info, and I just looked at this thing for the first time, so maybe it's brilliant and I'm ust not seeing it, but it sure looks a lot like they've fallen victim to a database noob mistake, and created a monster tree with disproportionate crazy branches everywhere, and that is bound to cause relational problems, redundant d

    • Maybe they're using the same company that cost the IRS a few hundred million a few years ago- for something that was eventually scrapped. With the current "it's fer terrism" mentality, and all the illegal spying and end-runs around various "obstacles" imposed by the constitution, I'd hope for a repeat performance.

  • War of the Worlds (Score:4, Insightful)

    by Lord Grey ( 463613 ) on Monday July 17, 2006 @06:29PM (#15734258)
    XML is, in part, supposed to make it easier to manipulate data by providing unambiguous definitions. It clarifies the data. So we throw the U.S. Government into the mix and wind up definitions like the following (pulled at random from the 'Definition' column within the niem-1.0beta.xls spreadsheet buried in the download):
    Authorized dissemination control portion mark abbreviation(s). Either (a) a single abbreviation or (b) a space-delimited list of abbreviations in the order shown in the CAPCO Register. Exception: For the REL abbreviation, omit the country code trigraph(s) and instead place the trigraph(s) in the releasableTo attribute value.
    WTF? This is perhaps a use of the word "Definition" that I am not acquainted with. It reminds me strongly of trying to decode the income tax rules while filling out those yearly forms. Possibly, those that actually understand the above will believe it to be a brilliant explanation. I guess I won't be one of those "lucky" contractors looking to implement NIEM-compliant software. Unless it's a "spook->human" translator.

    • Re:War of the Worlds (Score:3, Informative)

      by EQ ( 28372 )
      Actually that statment is pretty clear to someone with domain knowledge. Like any other knowlege domain, its probably very abstruse to the outside. Remember, Feynman was not famous for only being a physicist, but for being a physicist that could make himself understood to those outside of his domain of expertise (c.f. Feynman's lectures).

      Its actually a very concise and clear explanation of that part of the data plan. The problem for you is that you do not have the context, nor subject matter expertise, s
    • You write as if the government hasn't been in the business of data or high tech standards. This is light years better than EDI.
  • And there I was thinking TFA was about this:

    http://news.bbc.co.uk/1/hi/business/5187276.stm [bbc.co.uk]

  • I skimmed through the 58 page spec document which was mostly filled with describing the vast levels of bureaucracy that they're putting place to manage this beast. I also did a simple word find on the word "security". I only found 3 instances of the word that weren't coupled with the word "homeland" as in the Department of. No instances of the word "authentication".

    I know this is doc isn't intended to show the exact structure of the messages to be passed, but gee whiz, wouldn't you think they would add

  • >>Contact your local government contractor, with resume in hand, and you may be one of the lucky >>developers to implement NIEM-capable software. Lucky? When are good and decent programmers and other tech folks going to lay down their arms, so to speak, and refuse to further enable these turds? In an ideal world there would be a severe shortange of people who help enable the current US mal-administrations aims through technology. I'll starve before I help them commit any more crimes. Anyone
  • The most effective form of data sharing so far seems to have been the storage of that data on laptops leaving the building ...
  • XML is often poorly normalized such that you have to go out of your way to remove nesting, redundancies, etc. Why not clean up comma-delimited a bit so that it offers better meta-data (column types, etc.) and multi-schemas per file. Comma-delimited is also more compact. For some ideas, see:

    http://www.c2.com/cgi/wiki?RelationalAlternativeTo Xml [c2.com]
           
  • All the bloat will demand more storage, memory and faster cpus. With Vista late, XML is the only thing driving hardware sales these days. I used to say "XML is a solution looking for a problem". Now I understand the problem was flat hardware sales and XML is the solution.

    Yes, it will end up being used as database using XQuery, or worse custom implemetations of similar beasts.

    And yes, since it is not normalized data consistency will be lost leading to false psoitives and false negatives.

    And since it is so b
    • Meanwhile, the FBI is still running the story that their computer system is screwed up. They've been running this story for 10 years. Is that believable in the context of this rollout?
    • Having the NIEM guide database design is a danger. But, if an agency models their database on it, it won't because they weren't warned against it. (Not sure if there's the correct number of negatives there. The point is that agencies are being explicitly warned against using the NIEM as the basis for their databases.)

      Still, many make the initial assumption that the NIEM should guide their internal databases. Continual education is needed to prevent that misconception.

      Regarding stripped-down versions, there
  • I thought the government already had an "information sharing" program in place. Several of 'em, in fact:
    • This [cnn.com] (scroll down to "Other breaches revealed")
    • This [bbc.co.uk] (keep your stick safe!)
    • This [nytimes.com] (Yeah, I still hate the NYT... but even a bad example can be of some use...)
    • This [newsmax.com] (which has possible tie-in's to the previous...)
    • And, of course, this [google.com] (just to close with a "catch-all"...)

    Either way, never understimate the power of the government to screw something up .

  • Here's a summary of the major changes since version 0.9:
    • Boolean property niem.gov/niem/domains/intelligence/isPrisonerOfWa r has been replaced by enumeration { YES, NO, DEPENDS }
    • Likewise for boolean property niem.gov/niem/domains/immigration/isMexican
    • Namespace niem.gov/niem/domains/emergency-management/ now duplicates large portions of other namespaces - many functions appear to be documented but not implemented
    • Namespace niem.gov/niem/domains/justice/billOfRights has been deprecated in favor of nie

  • We are currently working with several government agencies that wish to expose data via the NIEM standard. The MetaMatrix product is being used to map current data sources into NIEM compliant views of that data without ever writing a line of code.

    We have a NIEM specfic example that demonstrates this capability by using a pre-fab Derby database. Our product is downloadable for a free trial for anyone who might be interested. Here is the link to the example:

    http://devcentral.metamatrix.com/products/ex [metamatrix.com]

Slashdot Top Deals

"More software projects have gone awry for lack of calendar time than for all other causes combined." -- Fred Brooks, Jr., _The Mythical Man Month_

Close