Skip to content

[JS] Timestamp types are all the same #91

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
asfimport opened this issue May 12, 2022 · 3 comments
Open

[JS] Timestamp types are all the same #91

asfimport opened this issue May 12, 2022 · 3 comments
Labels
Type: bug Something isn't working

Comments

@asfimport
Copy link

Current timestamp types are all the same. They have the same representation. And also the same precision.

For example, TimestampSecond and TimestampMillisecond return the values as 1652118180000. Instead, I would expect the TimestampSecond to drop the 3 zeros when returning a value, e.g. 1652118180. Also, the representation underneath is still an int32 array. Even though for TimestampSecond every second value is 0, the array still has double the amount of integers.

I also got an error when trying to read a Date as TimestampNanosecond - TypeError: can't convert 1652118180000 to BigInt.

Reporter: Teodor Kostov

Note: This issue was originally created as ARROW-16543. Please see the migration documentation for further details.

@asfimport
Copy link
Author

Paul Taylor / @trxcllnt:
[~terusus] how did you construct the Timestamp Vectors? The semantic meaning of the type is " since the epoch," so it's valid for the different duration dtypes to have the same underlying representation.

@asfimport
Copy link
Author

Teodor Kostov:
Hello [~paul.e.taylor]. I used a Struct type and a builder.

const myType = new arrow.Struct([
  new arrow.Field('time', new arrow.TimestampSecond()),
  new arrow.Field('value', new arrow.Float64()),
])
const myBuilder = arrow.makeBuilder({ type: myType, nullValues: [null, undefined] })
...
myBuilder.append(...)
...
myBuilder.finish()
const vector = myBuilder.toVector()

I am using another library that is sensitive to the timestamp format, whether in milliseconds or seconds. And for my application, I do not need the millisecond accuracy. At the same time, I would expect the data not to need any transformation when I get it from the table. I would also expect the table to use the most efficient format to store my data.

@asfimport
Copy link
Author

Teodor Kostov:
Here is a full test:

const dataType = new arrow.Struct([
  new arrow.Field('time', new arrow.TimestampSecond()),
  new arrow.Field('value', new arrow.Float64()),
])
const builder = arrow.makeBuilder({ type: dataType, nullValues: [null, undefined] })

const date = new Date()
const timestampSeconds = Math.floor(date.getTime() / 1000)
const timestamp = timestampSeconds * 1000
builder.append({ time: date, value: 1.2 })
builder.append({ time: date, value: 3.3 })
builder.finish()
const vector = builder.toVector()

const schema = new arrow.Schema(dataType.children)
const recordBatch = new arrow.RecordBatch(schema, vector.data[0])
const table = new arrow.Table(recordBatch)

console.log(timestamp)
console.log(timestampSeconds)
console.log(table.get(0).time)

console.log(table.get(0).time === timestamp) // should be false
console.log(table.get(0).time === timestampSeconds) // should be true

 

@assignUser assignUser transferred this issue from apache/arrow May 26, 2025
@assignUser assignUser added the Type: bug Something isn't working label May 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants