-
Notifications
You must be signed in to change notification settings - Fork 3
[JS] Timestamp types are all the same #91
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Paul Taylor / @trxcllnt: |
Teodor Kostov: const myType = new arrow.Struct([
new arrow.Field('time', new arrow.TimestampSecond()),
new arrow.Field('value', new arrow.Float64()),
])
const myBuilder = arrow.makeBuilder({ type: myType, nullValues: [null, undefined] })
...
myBuilder.append(...)
...
myBuilder.finish()
const vector = myBuilder.toVector() I am using another library that is sensitive to the timestamp format, whether in milliseconds or seconds. And for my application, I do not need the millisecond accuracy. At the same time, I would expect the data not to need any transformation when I get it from the table. I would also expect the table to use the most efficient format to store my data. |
Teodor Kostov: const dataType = new arrow.Struct([
new arrow.Field('time', new arrow.TimestampSecond()),
new arrow.Field('value', new arrow.Float64()),
])
const builder = arrow.makeBuilder({ type: dataType, nullValues: [null, undefined] })
const date = new Date()
const timestampSeconds = Math.floor(date.getTime() / 1000)
const timestamp = timestampSeconds * 1000
builder.append({ time: date, value: 1.2 })
builder.append({ time: date, value: 3.3 })
builder.finish()
const vector = builder.toVector()
const schema = new arrow.Schema(dataType.children)
const recordBatch = new arrow.RecordBatch(schema, vector.data[0])
const table = new arrow.Table(recordBatch)
console.log(timestamp)
console.log(timestampSeconds)
console.log(table.get(0).time)
console.log(table.get(0).time === timestamp) // should be false
console.log(table.get(0).time === timestampSeconds) // should be true
|
Current timestamp types are all the same. They have the same representation. And also the same precision.
For example,
TimestampSecond
andTimestampMillisecond
return the values as1652118180000
. Instead, I would expect theTimestampSecond
to drop the 3 zeros when returning a value, e.g.1652118180
. Also, the representation underneath is still anint32
array. Even though forTimestampSecond
every second value is0
, the array still has double the amount of integers.I also got an error when trying to read a
Date
asTimestampNanosecond
-TypeError: can't convert 1652118180000 to BigInt
.Reporter: Teodor Kostov
Note: This issue was originally created as ARROW-16543. Please see the migration documentation for further details.
The text was updated successfully, but these errors were encountered: