Skip to content

Number of staging files reported by API is incorrect #1149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
nitisht opened this issue Jan 30, 2025 · 0 comments · Fixed by #1151 or #1207
Closed

Number of staging files reported by API is incorrect #1149

nitisht opened this issue Jan 30, 2025 · 0 comments · Fixed by #1151 or #1207
Assignees
@de-sh
Labels
bug triage

Comments

@nitisht
Copy link
Member

nitisht commented Jan 30, 2025

Description:

We've observed that in cases where there is a very high number of files (.arrows / .parquet) in the staging directory, the number of files reported in the cluster page is incorrect. This requires investigation.
@de-sh de-sh mentioned this issue Jan 31, 2025
Merged
3 tasks
nikhilsinhaparseable added a commit to nikhilsinhaparseable/parseable that referenced this issue Feb 24, 2025
@nikhilsinhaparseable
fix: staging files count in metrics
701f050 
current - we find group of arrow files and a parquet file
in order to merge all arrow files in a group and write to parquet in disk
in metrics, we set count of arrow files for 1 group not all

change - we get sum of total count of arrow files from all the groups
then set this sum in metrics

fixes: parseablehq#1149
Merged
nitisht pushed a commit that referenced this issue Feb 25, 2025
@nikhilsinhaparseable
fix: staging files count in metrics (#1207)
505c314 
current - we find group of arrow files and a parquet file
in order to merge all arrow files in a group and write to parquet in disk
in metrics, we set count of arrow files for 1 group not all

change - we get sum of total count of arrow files from all the groups
then set this sum in metrics

fixes: #1149
7h3cyb3rm0nk pushed a commit to 7h3cyb3rm0nk/parseable that referenced this issue Feb 26, 2025
@nikhilsinhaparseable @7h3cyb3rm0nk
fix: staging files count in metrics (parseablehq#1207)
e9bc70a 
current - we find group of arrow files and a parquet file
in order to merge all arrow files in a group and write to parquet in disk
in metrics, we set count of arrow files for 1 group not all

change - we get sum of total count of arrow files from all the groups
then set this sum in metrics

fixes: parseablehq#1149
7h3cyb3rm0nk pushed a commit to 7h3cyb3rm0nk/parseable that referenced this issue Feb 26, 2025
@nikhilsinhaparseable @7h3cyb3rm0nk
fix: staging files count in metrics (parseablehq#1207)
d854f78 
current - we find group of arrow files and a parquet file
in order to merge all arrow files in a group and write to parquet in disk
in metrics, we set count of arrow files for 1 group not all

change - we get sum of total count of arrow files from all the groups
then set this sum in metrics

fixes: parseablehq#1149
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@de-sh de-sh

Labels
bug triage
Projects
None yet
Milestone
No milestone
3 participants
@nitisht @de-sh @nikhilsinhaparseable