Comment On Interesting Bitmap

Another steel mill story? wow... and my senior project was related to this (the steel mill stuff, not the signature handling XMLAOTD (XML Abuse Of The Day)).

Note: my senior project mostly caught fire. Thermal runaway in motor drivers is my worst nightmare.

XML meta: ~50 chars
payload: 2 to 3 chars
6% payload;

The real WTF is that they didn't use JSON!

Ah but seriously, what's with the two layers of tags?
Someone was itching to use this new-fangled XML stuff.

Alex:

<Signature
<signature>
87
</signature>
</Signature>
<Signature>
<signature>
127
</signature>
</Signature>

Well there's your problem. You're not closing the initial <Signature> tag.

>> And not the "everyday" things like the gargantuan ladles that carry hundreds of tons of molten steel...

It took me a second to realize that didn't say gargantuan ladies.

Must remember to have coffee before reading.

Ummm, that's not X Y data. At best it might be the value of each consecutive pixel, with the X Y coordinates implied. But, you said it wasn't a color image.

I call BS. Or confusion, which is close enough to the same thing.

I would've expected
<signature>
<pixel>
<x>3</x>
<y>53</y>
</pixel>
...
</signature>

I don't even understand how they use this stuff or what crazy thoughtprocess made this come into existence.

(Yeah I do understand that using xml for this is a total wtf)

@Deprecated:

XML meta: ~50 chars
payload: 2 to 3 chars
6% payload;

The real WTF is that they didn't use JSON!

Ah but seriously, what's with the two layers of tags?
Someone was itching to use this new-fangled XML stuff.

the Real WTF is that they didn't use Ice (ZeroC) or it's crappier ancestor CORBA.

but JSON would have been ok was well...

for an ICE payload it would be very little overhead + the binary size of the array. overhead is fixed. O+N*sizeof(typeof(N))

for a JSON payload [assuming 2 byte average per datapoint] 2 + 2*N + N where 2*N characters of that are payload and 2+N are overhead. if you don't pack it with spaces (Which are unneeded since it is generated data anyway)

Man of Steel:

Ummm, that's not X Y data. At best it might be the value of each consecutive pixel, with the X Y coordinates implied. But, you said it wasn't a color image.

I call BS. Or confusion, which is close enough to the same thing.

you can address an X-Y space with consecutive numbers if you map it correctly

N = the value given
Sx = Width
Sy = Height
Origin = topleft.

X = int(N / Sx)
Y = N % Sx

IIRC.. i tried to remember that real fast.. might be in error.

Roy:

I would've expected
<signature>
<pixel>
<x>3</x>
<y>53</y>
</pixel>
...
</signature>

I don't even understand how they use this stuff or what crazy thoughtprocess made this come into existence.

(Yeah I do understand that using xml for this is a total wtf)

Geez, whatta XMLn00b.

The best, most clearly-normalized format would have to be
<signature>
<pixel x=0 y=0>
0
</pixel>
<pixel x=1 y=0>
0
</pixel>
<pixel x=2 y=0>
1
</pixel>
<pixel x=3 y=0>
0
</pixel>
...
</signature>

Really.

The basic rule of XML is "XML is like violence. If it isn't solving your problem yet, you're not using enough of it."

No, no, no, if you want to waste bytes, put your X Y data in VRML!

#VRML V2.0 utf8



Shape {

 appearance Appearance {

  material Material {

   emissiveColor 0.8 0.8 0.8

  }

 }

 geometry PointSet {

  coord Coordinate {

   point [

    3 17 0

   ]

  }

 }

}





NavigationInfo {

 type [ "EXAMINE", "FLY", "WALK", "ANY" ]

 speed 1.0

}

I mis-read "gargantuan ladles" in the first paragraph as "gargantuan ladies". Then got all excited when it started talking about the "hot strip mill"...

Creating a hostile workplace environment aside, this story is an excellent example of why XML is not always your friend.

XML meta: ~50 chars
payload: 2 to 3 chars
6% payload;

That's only if you're still thinking like the developer who wrote the original. This is a *bitmap*. A black and white bitmap, at that. Let's be charitable and say that the signature field is 400px by 300px. That's 1200 bits of data, or 150 bytes. Just shoot that binary data at the bitmap creator and you're golden.

Compare that to 50 bytes per block times 1000 such blocks for 50kB of data, and your actual payload is .3%.

Even with full 32-bit color, they could have stored it with 4 bytes per block. The size of the XML at that point, though...

(Also, had they used JSON, they could have serialized the whole thing as "[127,43,54,...]" and gotten a ~75% payload.)

ContraCorners:

>> And not the "everyday" things like the gargantuan ladles that carry hundreds of tons of molten steel...

It took me a second to realize that didn't say gargantuan ladies.

Must remember to have coffee before reading.

Me too. Good for a smile, though.

The uncompressed bitmap data transmitted through the webservice weighed in at about 290KB.

The resulting XML looked like this...

So XML is "uncompressed bitmap data" now?

TRWTF is the contradictions in this story, which can be summarized as "Past developer used XML badly." You're welcome, Daily WTF. I just saved you 20KB in "uncompressed bitmap data" (or "characters" if we're being all LOGICAL).

Kazan:

Man of Steel:

Ummm, that's not X Y data. At best it might be the value of each consecutive pixel, with the X Y coordinates implied. But, you said it wasn't a color image.

I call BS. Or confusion, which is close enough to the same thing.

you can address an X-Y space with consecutive numbers if you map it correctly

N = the value given
Sx = Width
Sy = Height
Origin = topleft.

X = int(N / Sx)
Y = N % Sx

IIRC.. i tried to remember that real fast.. might be in error.

The examples given, 87 and 127, don't contain enough bits to encode X and Y in a single number, unless the resulting bitmap is very small. Well I suppose X could have been 0 in both cases and the signature line went from Y=87 to Y=127 but that seems improbable for a signature capture.

Presumably the thinking was this:

"Signatures...Hmmmm...Well, in handwriting analysis, you don't just look at the picture, you actually look and see whether they wrote it in the right order (like they started from the top of the letter vs from the bottom). It's true, I saw it on CSI last night. We'd better store each individual pixel they go over, in order, so that we can check that it's not a forgery."

Never mind that the requirement was just to store the images. And they would never need to check for forgeries. And they didn't have an algorithm to detect forgeries anyway.

On that note, the real wtf is that they didn't store pressure data which is effective for detecting forgeries!

"And not the "everyday" things like the gargantuan ladles that carry hundreds of tons of molten steel."

I read this as "gargantuan ladies"

<Comment>
<comment>
I ain't first
</comment>
</Comment>

I think a lot of us in the wee hours of the morning misread that as "ladies" Me included.

BBT:

Presumably the thinking was this:

"Signatures...Hmmmm...Well, in handwriting analysis, you don't just look at the picture, you actually look and see whether they wrote it in the right order (like they started from the top of the letter vs from the bottom). It's true, I saw it on CSI last night. We'd better store each individual pixel they go over, in order, so that we can check that it's not a forgery."

Never mind that the requirement was just to store the images. And they would never need to check for forgeries. And they didn't have an algorithm to detect forgeries anyway.

On that note, the real wtf is that they didn't store pressure data which is effective for detecting forgeries!

No, I'm guessing it's more likey the original dev was just plain stupid.

Honestly, binary data is not exactly black magic...

And if you absolutely had to send it via a webservice, wouldn't a binary protocol like Hessian make this trivial?

XML Blasta:

Geez, whatta XMLn00b.

The best, most clearly-normalized format would have to be
<signature>
<pixel x=0 y=0>
0
</pixel>
<pixel x=1 y=0>
0
</pixel>
<pixel x=2 y=0>
1
</pixel>
<pixel x=3 y=0>
0
</pixel>
...
</signature>

Really.

The basic rule of XML is "XML is like violence. If it isn't solving your problem yet, you're not using enough of it."

How's about
<signature>
<row y=0>
<pixel x=0>
0
</pixel>
<pixel x=1>
1
</pixel>
<pixel x=2>
0
</pixel>
...
</row>
<row y=1>
<pixel x=0>
0
</pixel>
...
</row>
...
</signature>

Xanthir, FCD:

XML meta: ~50 chars
payload: 2 to 3 chars
6% payload;

That's only if you're still thinking like the developer who wrote the original. This is a *bitmap*. A black and white bitmap, at that. Let's be charitable and say that the signature field is 400px by 300px. That's 1200 bits of data, or 150 bytes. Just shoot that binary data at the bitmap creator and you're golden.

Compare that to 50 bytes per block times 1000 such blocks for 50kB of data, and your actual payload is .3%.

Even with full 32-bit color, they could have stored it with 4 bytes per block. The size of the XML at that point, though...

(Also, had they used JSON, they could have serialized the whole thing as "[127,43,54,...]" and gotten a ~75% payload.)

Well, first, I've obviously just woken up, because 400x300 is 120000 bits, or 15000 bytes. 15k. This makes my experimental evidence (a 400x300 monochrome bitmap produced in MS Paint) make sense, as it's 15662 bytes in size.

And, rereading the article, I see that the signatures are actually 300x100. So 30000 bits, 3750 bytes. Thus, this approach is pretty comparable to recording just the black bits in a sparse array. Given a decent packed encoding (number the pixels consecutively, so you can store them in 2 bytes), and 1000 black pixels, you've got the entire thing stored in 2kB.

Intelligent choice of data structures ftw!

ContraCorners:

>> And not the "everyday" things like the gargantuan ladles that carry hundreds of tons of molten steel...

It took me a second to realize that didn't say gargantuan ladies.

Must remember to have coffee before reading.

Hah! You did better than me. I thought it said ladies until I read your comment. :P

Fahrenheit? Speak SI, dammit!

Prometheus:

<Comment>
<comment>
I ain't first
</comment>
</Comment>

I think a lot of us in the wee hours of the morning misread that as "ladies" Me included.

Still fun to visualize, though... that's the true essence of a WTF!

anon:

Fahrenheit? Speak SI, dammit!

Sports Illustrated?

When I first read that the 300x100 black and white image took up 290kb, I said to myself in my head "Haha, what, are they storing it in XML or something?"

I'm not laughing any more.

anon:

Fahrenheit? Speak SI, dammit!

2000 degrees Fahrenheit is approximately "a billion degrees" in Celsius and "a billion - 273" Kelvin.

another anon:

anon:

Fahrenheit? Speak SI, dammit!

2000 degrees Fahrenheit is approximately "a billion degrees" in Celsius and "a billion - 273" Kelvin.

If billion === thousand, and 273 == -273.

Oh god, 273 == -273. The universe goes boom.

Roy:

I would've expected
<signature>
<pixel>
<x>3</x>
<y>53</y>
</pixel>
...
</signature>

I don't even understand how they use this stuff or what crazy thoughtprocess made this come into existence.

(Yeah I do understand that using xml for this is a total wtf)

The thing is, you could just as easily do:
<signature>
<pixel>
3,53
</pixel>
...
</signature>

...And cut your xml in half. To be even crazier, you could remove the xml entirely and just do a big comma separated list: x,y,x,y,x,y,x,y,etc.

Really though, you should just offload the work to the data input device and get it to send you a fricking jpeg. This isn't 1981.

Ah, XML. The cause of, and solution to, all of life's problems.

Xanthir, FCD:

Xanthir, FCD:

XML meta: ~50 chars
payload: 2 to 3 chars
6% payload;

That's only if you're still thinking like the developer who wrote the original. This is a *bitmap*. A black and white bitmap, at that. Let's be charitable and say that the signature field is 400px by 300px. That's 1200 bits of data, or 150 bytes. Just shoot that binary data at the bitmap creator and you're golden.

Compare that to 50 bytes per block times 1000 such blocks for 50kB of data, and your actual payload is .3%.

Even with full 32-bit color, they could have stored it with 4 bytes per block. The size of the XML at that point, though...

(Also, had they used JSON, they could have serialized the whole thing as "[127,43,54,...]" and gotten a ~75% payload.)

Well, first, I've obviously just woken up, because 400x300 is 120000 bits, or 15000 bytes. 15k. This makes my experimental evidence (a 400x300 monochrome bitmap produced in MS Paint) make sense, as it's 15662 bytes in size.

And, rereading the article, I see that the signatures are actually 300x100. So 30000 bits, 3750 bytes. Thus, this approach is pretty comparable to recording just the black bits in a sparse array. Given a decent packed encoding (number the pixels consecutively, so you can store them in 2 bytes), and 1000 black pixels, you've got the entire thing stored in 2kB.

Intelligent choice of data structures ftw!

I've dealt with too much COBOL in my time to try and optimize the data storage for size in all but the most extreme circumstances. Size and Cycles are cheap; the data structure needs to be optimized for speed and reliability.

Not that this does that either. But the size isn't what bothers me about it.

Not that they did this the right way at all, but remember that they were trying to capture signatures here. Not just images of signatures. Things like the direction a letter is drawn might be checked if you have the consecutive point data rather than a monochrome bitmap.

So, you're saying, if I notice my developers working out calculus functions on a white board while trying to figure out how to store a signature bitmap...I should be concerned?

Sometimes, I like to insert this type of problem solving technique into other professions in my mind, such as a doctor!

"Sure, we could remove the splinter in your index finger with tweezers, but I just got all this fancy new laproscopic equipment and I've been dying to try it out! Let's try getting at it through the belly button!"

Man of Steel:

The examples given, 87 and 127, don't contain enough bits to encode X and Y in a single number, unless the resulting bitmap is very small. Well I suppose X could have been 0 in both cases and the signature line went from Y=87 to Y=127 but that seems improbable for a signature capture.

or it (the bit shown int he article) is a snippet of a sparse array

The Code — the lifeblood of the steel mill's information system applications — was originally developed by a highly paid consulting firm who specialized in the manufacturing industry and hired "only the brightest of developers".

Whenever you hear or read something like this, you know that things are going to get very, very ugly.

Satanicpuppy:

Really though, you should just offload the work to the data input device and get it to send you a fricking jpeg. This isn't 1981.

Ok, firstly: a lossy image format for signature data? You're nuts, friend.

Secondly, the web service probably required XML input (that's how these requirements go: "thou shalt use SOAP and XML for the webservice"), so "just sending an image" wouldn't be a great solution.

There's also the need to think ahead about future requirements that might change the image size or require color (stupider regulations have been passed).

So here's what I would do:

A 300x100 monochrome image is just an array of 30000 bits. I would have packed this into 3750 bytes, then Base64 encoded the whole thing, and slipped it into the webservice payload thus:

<signature width="300" height="100" color="no">
<image encoding="base64">....encdoded data here...</image>
</signature>

XML Blasta:

Geez, whatta XMLn00b.

The best, most clearly-normalized format would have to be
<signature>
<pixel x=0 y=0>
0
</pixel>
...
</signature>

Really.

The basic rule of XML is "XML is like violence. If it isn't solving your problem yet, you're not using enough of it."

Heh,
+1 internets for proper use of XML.

In a similar vein I worked on a job where we wanted to punt around large amounts of numerical data from a Hot Strip Mill setup program to a logging system.

We proposed an ASN.1 solution that would minimise the data volume and was easily extensible if you wanted to add more data to the log.

But did they go with that?? Nope .. they ended up with an XML solution which multiplied the dataset size by a huge amount and if you wanted to change the dataset you had to edit and recompile the setup program.

Doesn't seem like a bad idea, very enterprisy in my opinion. At least the won't have a hard time comparing signatures... oh wait!

Why do we need to pass it to the server if it's base64?

Darren:

Satanicpuppy:

Really though, you should just offload the work to the data input device and get it to send you a fricking jpeg. This isn't 1981.

Ok, firstly: a lossy image format for signature data? You're nuts, friend.

JPEG doesn't have to be lossy, does it? I thought at 100% in certain encoders it is lossless.

At any rate, just stick the damn thing in as a PNG or a GIF or something. I don't think JPEG is very efficient in monochrome (although I could be wrong).

ObiWayneKenobi:

The Code — the lifeblood of the steel mill's information system applications — was originally developed by a highly paid consulting firm who specialized in the manufacturing industry and hired "only the brightest of developers".

Whenever you hear or read something like this, you know that things are going to get very, very ugly.

Yeah, but it builds suspense so much better than, "...a reasonably paid consulting firm with no particular specialty who hired average developers."

Dinnerbone:

Oh god, 273 == -273. The universe goes boom.

That makes perfect sense iff 273 == i

evilspoons:

At any rate, just stick the damn thing in as a PNG or a GIF or something. I don't think JPEG is very efficient in monochrome (although I could be wrong).

You're not wrong. JPEG is really freakin' awful for monochrome images. It's also pretty bad at any line-art like a cartoon or logo, color or mono. JPEG compresses smooth color gradients, not sharp transitions. Encoding a line drawing as JPEG gives it all sorts of really nasty compression artifacts.

JPEG would be the worst possible image format for a monochrome signature. With the possible exception of XML, of course.

Dinnerbone:

another anon:

anon:

Fahrenheit? Speak SI, dammit!

2000 degrees Fahrenheit is approximately "a billion degrees" in Celsius and "a billion - 273" Kelvin.

If billion === thousand, and 273 == -273.

http://www.google.com.au/search?q=2000f+in+c

But the second part is "a billion - 273" as in 999999727 "another anon Kelvin".

XML is a virus.

This sounds like the app I worked on at my previous job. We bought a company that made remote inspection software, and it was full of crap like this. It sounds *exactly* like Echo by Infowave (previously Telispark). The guy who wrote the original did some crazy stuff like storing strokes of the signature instead of storing the binary bitmap.

I would not at all be surprised if this was Telispark...

Darren:

So here's what I would do:

A 300x100 monochrome image is just an array of 30000 bits. I would have packed this into 3750 bytes, then Base64 encoded the whole thing, and slipped it into the webservice payload thus:

<signature width="300" height="100" color="no">
<image encoding="base64">....encdoded data here...</image>
</signature>

Thats no fun - I would have converted it to ascii art.

Jim Lard:

I mis-read "gargantuan ladles" in the first paragraph as "gargantuan ladies". Then got all excited when it started talking about the "hot strip mill"...

I'm a gargantuan lady, you insensitive clod!

It must be Friday. I read the second sentence as 'And not the "everyday" things like the gargantuan ladies that carry hundreds of tons of molten steel.'