Handling integers

[hierozbosch]

The handling of integers is referenced in previous issues (153, 122, 106, etc.). These issues have been closed and point to the Bolt driver methods (InSafeRange, int, etc.) as the resolution. However, using these methods implies hard-coding conversions against a known set of data entities, and doing it on the server. It's a highly inefficient solution: the driver should only convert 64-bit numbers to {high: low:} objects as a last resort. Converting all integers into these objects and then requiring a server to locate and re-convert most of them back into JS numbers is not an ideal solution. I'm using the neo4j-javascript-driver for a web app, and I would be very surprised if serious 64-bit applications considered JS their language of choice. In my case: a.) all my numbers are JS numbers, both in and out of neo4j; b.) I do not want to use my back-end neo4j server and front-line web servers to convert JS integers into objects and then convert those objects back into JS integers; and c.) I do not want to hard-code my server or client scripts to the granularities of a data model. I'm raising this as a new issue: I think the neo4j-JS-driver should have an optional directive to convert 64-bit integers directly to JS numbers and not waste time converting them all into objects. Thanks!

[lutovich]

Hi @hierozbosch,

Thanks for reporting this.

I feel that converting to Integer objects as a last resort might be strange. Some numbers will then be regular javascript numbers and some will be Integers which could be even more confusing. You are right saying that we could have a switch like {useNativeNumbersOnly: boolean} but what should driver do if it receives a 64-bit number from the database? It is possible for multiple applications written in different languages to talk to the same database. Stored numbers could have any number of bits in such case.

All those mentioned issues and this one definitely show that Integer class with {high: ..., log: ...} does not feel very native and convenient. We'll think about improving number handing in upcoming releases.

[hierozbosch]

Thanks Lutovich, Just by way of background, I've been running on neo4j for 5 years, since v 0.7. We use the graph for supply chain management--relationship tracking, access control, and social media stuff. We deal in smallish numbers--JS dates are probably the only time we exceed the ^32 range, and we don't see a case where we'd get over 2^53 with this app.

I put off the upgrade to neo4j 3.x for over a year due to the breakages, but I'm deep in it now. I had to change a lot of cypher, and almost all the result-handling. That's when the issue with the integers came up.

Looking at the driver sources briefly this a.m., it appears the data stream is coming into the driver with the integers already cut before it hits the 'packables' methods.

I guess what I'd suggest would be an optional flag that a user could inject when making the call to the driver, like: neo4j.run( query, params, JS_NUMBERS) ...

The packer would check that flag; if set, it would just pack and return a JS number. If it hits a big number with the flag set, have it throw an error-- INTEGER_OUT_OF_RANGE.

For some users, like me, that would be enough--not likely I would ever see that error, but if I did I would just remove the flag from that query and build in the number handling on a case-by-case basis.

That might help some users make the upgrade to Bolt easier. I'm already watching for errors on every query, of course, so no change there. I could get up and running without having to go through the entire code base looking for every instance where an integer comes out of the db, as I'm in the process of doing now.

Thanks!

[bryanph]

I think a common use case is writing only through the javascript-driver anyway. So any overflows will not occur, because there are no external writes causing integers bigger than 2^53.

You might also just simply return a string with the integer value and probably cause less memory then repeatedly initiating an Integer class. The Twitter API for example, simply provides an "id_str" field to account for overflows. This is why I proposed an option for simply returning string representations instead of instances of Int in #106

[hierozbosch]

Thanks bryanph, yes, that's my use-case, too: all read/write via javascript-driver; therefore, no overflows above 2^53. I prefer an option on the query just to give me JS numbers--in fact I've already modified the driver to do this (https://stackoverflow.com/questions/44116453), but it would be better to have it supported. I'm not sure I understand the need for a "one-size-fits-all" on this particular issue, when neo4j offers options on many other things. Make it default to 64-bit if most people want that, so it's transparent for them.

[bryanph]

@lutovich @hierozbosch I don't mind casting the string to a javascript integer at a later time. In this case, you put the responsibility in the users hands. The user can check if the string represents a legal integer and do the casting explicitly whenever they want to display or use the string as a number. Or the user could use a library of their choice. In my opinion there is no good reason for this library to implement its own Integer class. It simply makes the library more difficult to use.

[bryanph]

I forked the repo and added an option "convertToString" to always return strings instead of a neo4j.int object. See bryanph@9f9d644

[lutovich]

Hi @hierozbosch, @bryanph.

Please check #323 which makes it possible to create driver that only exposes native numbers.

[hierozbosch]

Nice. I'll check it out and get back to you, but looks like a good fix for my purposes.

--- Matthew Hodges GeoCertify, LLC e: [email protected]

On 2018-01-17 06:39, Konstantin Lutovich wrote: Hi @hierozbosch [1], @bryanph [2]. Please check #323 [3] which makes it possible to create driver that only exposes native numbers. -- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [4], or mute the thread [5].

Links: ------ [1] https://github.com/hierozbosch [2] https://github.com/bryanph [3] #323 [4] #245 (comment) [5] https://github.com/notifications/unsubscribe-auth/AEF2n2GxljMWfOJkbHCVf_q03k9Hzi8Uks5tLdv2gaJpZM4NioHZ

[lutovich]

#323 is now merged. Change will be part of 1.6.0-alpha01.