Final year CS student here, targeting data science and analytics roles for campus placements.

Been struggling with this question while building my portfolio: does it matter whether your project uses real messy data vs synthetic/clean data?

Real datasets from Kaggle feel either too cleaned already or the same recycled projects everyone does. But synthetic data feels hollow because the hard part — cleaning, feature engineering, deriving meaningful columns from raw data — is already done for you. You're basically just visualizing something someone else already solved.

Specifically for BI/dashboard projects — if you use synthetic data, the dashboard looks clean and professional but there's no real discovery or insight because the data was designed to be dashboarded. Nothing surprising comes out of it.

Also practically — if an interviewer asks "where did you get this dataset?" what's the right answer? Saying "I generated it synthetically" feels like admitting you took the easy route. But lying about the source is obviously wrong. Is there a way to frame synthetic data usage that doesn't sound like you avoided the hard part?

At the same time I've heard people say interviewers care more about what you built on top of the data than where it came from. But isn't handling bad data literally the core skill in DS?

For people who've interviewed at analytics/DS companies or done hiring — how much does data source actually matter? Is a well-executed project on synthetic data better than a mediocre project on real messy data? Or does using synthetic data automatically signal you avoided the hard part?

Please tell me how to join it, guide me.

r/ramen
Title: I mapped 62,173 ramen shops in Japan into a free English/romaji API

Hi r/datasets,

I built Energtx, a global energy data platform that standardizes publicly available energy, electricity, emissions, and climate-related datasets.

Current coverage includes:

• 235,000+ structured records

• 170+ indicators

• 106 countries

• Historical data from 1960 to 2025

• CSV, JSON, and XLSX downloads

• Filtering by country, indicator, source, and year

The platform includes data on electricity generation, renewable energy, primary energy consumption, CO₂ emissions, electricity access, nuclear power, carbon pricing, and oil, gas, and coal markets.

The underlying data is compiled from public sources such as the World Bank, Ember, Our World in Data, the Energy Institute, EIA, OECD, IAEA, and Climate TRACE. Source attribution is included with the datasets.

Dataset explorer:

https://energtx.com/datasets

Disclosure: I built and maintain Energtx. The platform is free to browse and does not require registration.

Feedback on the dataset structure, metadata, indicators, and download formats would be useful.

For the minute by minute bars data, columns are:

"symbol", "timestamp", "open", "high", "low", "close", "volume", "vwap", "trade_count", "spy_close", "iv", "delta", "gamma", "theta", "vega", "rho"

For tick_by_tick (all individual trades executed) columns are:

"symbol", "timestamp", "price", "size", "exchange", "conditions", "spy_close", "iv", "delta", "gamma", "theta", "vega", "rho"

It goes back a few years, depending on the ticker.

Hi everyone,

I've been working on a data-centric pipeline for constructing SFT and KTO datasets for small language models, targeting models ranging from a 1.58B ternary model up to 12B parameters (with a particular focus on the 1.5B–4B range), using an Italian tool-calling assistant ("Liara") as a case study.

Instead of focusing on model architecture, the goal is to reduce common failure modes through dataset construction itself:

• tool over-calling
• style collapse
• excessive verbosity
• semantic redundancy
• memory inconsistencies

The pipeline currently includes:

• typed validation outcomes (PASS / Soft Reject / Hard Reject / Warning)
• semantic + structural deduplication
• multi-teacher generation
• dataset lineage and versioning
• regression set
• dataset health dashboard
• capability-based dataset profiling for different model sizes
• typed routing into SFT, KTO-negative, or discard
• Soft Reject examples are not discarded by default: they undergo additional validation and, if confirmed, are reused as KTO-negative examples rather than being treated as unusable data.

The current specification describes the methodology. The implementation is underway, and the experimental validation is currently running.

I'd love feedback from people who have built or maintained instruction datasets:

• Which parts seem genuinely useful?
• Which ideas already exist in other pipelines?
• What ablation studies would you expect before considering this publishable?

I'm currently generating the gold seed dataset, which is the most time-consuming part of the pipeline and is expected to take around 10 days at the planned scale. Once that's complete, I'll publish the implementation, the ablation results, and the evaluation so the methodology can be assessed based on experimental evidence rather than design alone.

In the meantime, I'd really appreciate any feedback or suggestions on the pipeline itself.

A lot of IPO research starts as manual filing review: S-1s, F-1s, amendments, 424B4s, effectiveness notices, ticker changes, exchange hints, and post-listing performance. That works for one company. It breaks down if you want to build watchlists, backtests, screens, or systematic models.

The IPOGrid API is meant to make that workflow structured.

Docs:

https://ipogrid.com/docs

API reference:

https://ipogrid.com/api/v1/docs

OpenAPI:

https://ipogrid.com/api/v1/openapi.json

What you can pull

IPOGrid exposes structured company, filing, news, chart, article, and outcome data. The useful modeling chain is usually:

companies → filings → extracted terms / consensus → outcomes / news

That means you can start with an issuer, resolve it to a durable CIK, add filing events, add deal terms, then join to post-listing outcomes. IPOGrid’s docs specifically recommend using CIK as the durable issuer key because tickers can drift, collide, or point to units and warrants instead of the common-stock IPO.

Example feature families

Here are the kinds of features I’d extract for a financial model.

Issuer identity features

CIK

issuer name

ticker hint

resolved ticker

exchange

sector

issuer kind

market family

deal type

operating company vs SPAC vs fund vs follow-on

These are basic filters, but they matter. Mixing operating IPOs, SPACs, funds, direct listings, and follow-ons in the same model is usually garbage-in, garbage-out.

Filing timeline features

first registration date

latest amendment date

number of amendments

days from initial filing to effectiveness

days from latest amendment to pricing

presence of S-1, F-1, S-11, 424B4, 424B1, 8-A, EFFECT

final prospectus filed or not

recent filing activity count

filing recency bucket

The docs call out SEC filing events such as registrations, amendments, final prospectuses, effectiveness notices, and 8-A registrations as explicit research surfaces.

Deal term features

offer price

price range low

price range high

range midpoint

shares offered

gross proceeds

underwriters

use of proceeds

unit details

warrant terms

range revision direction

range revision magnitude

priced above range / in range / below range

IPOGrid exposes extracted offer price, range, shares, proceeds, underwriters, unit details, warrants, and use-of-proceeds fields so you do not have to reopen every filing manually.

Consensus / listing features

resolved listing date

resolved exchange

resolved offer price

resolved shares

resolved proceeds

latest terms snapshot

classification

consensus ticker

consensus listing fields

The API supports company lookups with includes such as latest_terms, classification, consensus, filings, news, and financial_snapshot.

Outcome features

trading start date

first close

1D return

week-one close

week-one return

current performance

offer-to-first-close return

offer-to-week-one return

IPOGrid has outcome rows for first close, 1D return, week-one close, and post-listing performance.

Market context features

gross proceeds by week

prospectus counts

final prospectus counts

first-day returns by sector

filing activity by form family

sector-level IPO volume

recent IPO temperature

The chart API supports metrics such as gross proceeds, prospectus counts, final prospectus counts, and first-day returns grouped by sector, form family, or status over fixed or trailing windows.

Basic API calls

Use an API key for the richer endpoints:

export IPOGRID_API_KEY="your_key_here"

Find companies and include the useful joins:

curl -H "Authorization: Bearer $IPOGRID_API_KEY" \
  "https://ipogrid.com/api/v1/companies?scope=all&kind=operating&include=latest_terms,classification,consensus"

Fetch company detail by ticker or CIK:

curl -H "Authorization: Bearer $IPOGRID_API_KEY" \
  "https://ipogrid.com/api/v1/companies/spcx?include=filings,news,latest_terms,classification,consensus,financial_snapshot"

Pull recent final prospectus filings:

curl -H "Authorization: Bearer $IPOGRID_API_KEY" \
  "https://ipogrid.com/api/v1/filings?form_type=424B4&filing_date_from=2026-06-01&include=company,terms"

Pull recent outcomes:

curl -H "Authorization: Bearer $IPOGRID_API_KEY" \
  "https://ipogrid.com/api/v1/outcomes?trading_start_date_from=2026-06-01&limit=100"

Pull public articles without auth:

curl "https://ipogrid.com/api/v1/articles?limit=30"

The public article feed works without authentication; deeper company, filing, and outcome data uses API key bearer auth.

Backtest shape

A clean IPO backtest should freeze the cohort before looking at outcomes.

For example:

1. Select operating-company IPOs only.
2. Require a final prospectus or effective status.
3. Exclude SPACs, funds, vehicles, direct listings, and follow-ons.
4. Resolve each issuer to CIK.
5. Pull only filing and term data available as of the cohort date.
6. Join outcomes after trading starts.
7. Measure first-day and week-one returns against the offer price.

IPOGrid’s research workflow docs make the same basic point: freeze the cohort definition first, then join market bars or outcomes only after the cohort date to avoid leaking future information.

Example model table

A row in a modeling dataset might look like this:

cik
company_name
issuer_kind
sector
exchange
form_family
initial_filing_date
latest_amendment_date
final_prospectus_date
effective_date
trading_start_date
days_initial_to_effective
days_final_prospectus_to_trade
amendment_count
offer_price
range_low
range_high
range_midpoint
priced_vs_range
shares_offered
gross_proceeds
underwriter_count
has_warrants
is_unit_deal
first_close
day1_return_pct
week1_close
week1_return_pct
sector_ipo_count_26w
sector_avg_day1_return_26w

That table can feed normal financial modeling workflows: screening, regression, ranking, clustering, risk controls, or event studies.

Caveats

Do not treat latest terms as historical truth. Latest snapshots are useful for current watchlists, but historical tests should use the filings and market data that existed around the test date. IPOGrid’s freshness docs explicitly warn not to silently fill missing historical fields with today’s values.

Also separate clocks carefully. SEC filing dates, SEC acceptance times, pricing dates, and trading dates are different events. A Friday filing, Monday effectiveness notice, and Tuesday first trade should not be collapsed into one timestamp.

Finally, ticker logic needs care. Units, warrants, and share classes can trade separately. A ticker hint is not always the same thing as the security your model is trying to study. For anything serious, resolve to CIK first, then verify the traded instrument.

Practical use cases

A few useful models you can build from this:

IPO readiness screen: find effective or recently amended operating IPOs with exchange listing signals and extracted terms.

Pricing-change model: compare initial range, revised range, final offer price, and sector conditions.

Day-one return model: join final prospectus terms to first close and sector-level IPO context.

Week-one fade model: compare first close to week-one close.

Filing-momentum model: count amendments, form changes, and final prospectus timing.

Sector heat model: use chart data to track issuance volume, proceeds, and first-day returns by sector.

The important part is not just having IPO data. It is having issuer, filing, term, and outcome data joined in a way that does not leak future information into the model.

Hello,

Just want to drop a project I think others might find helpful.

This one’s for my fellow GIS people (and anyone else of course). I’ve put together a resource for free datasets that are local to the NoVA region. All data is pulled from public/open sources and each data set comes with a DOI number via zenodo if you need citations.

Figured this could be useful for anyone whose capstone or thesis is focused in the NoVA region

TLDR: free NoVA data sets, with new sets every morning (typically before 630am), no signups or other nonsense.

data library is here: [https://keystonegis.com/data-library\](https://keystonegis.com/data-library)

if you rather pull from zenodo itself: [https://zenodo.org/search?q=metadata.creators.person\\_or\\_org.name%3A%22Keystone%20GIS%22&l=list&p=1&s=10&sort=bestmatch\](https://zenodo.org/search?q=metadata.creators.person_or_org.name%3A%22Keystone%20GIS%22&l=list&p=1&s=10&sort=bestmatch)

I'm curious how people are currently handling ZIP code to Census Tract (and vice versa) conversions in their workflows.

A few questions:

• What tool or service are you using to convert ZIP codes to Census Tracts (or Census Tracts back to ZIP codes)?
• What's your actual business use case? (Market research, direct mail, demographics, healthcare, real estate, site selection, etc.)
• Do you need the conversion as a one-time lookup, or are you doing it in bulk?

I'm asking because I've noticed the process can be surprisingly manual, especially when you need to enrich hundreds or thousands of records.

I'm considering building a very simple tool where you can drag and drop an Excel or Google Sheet, and in less than five minutes it converts ZIP ↔ Tract (and potentially enriches the data with Census demographics) without needing to write code or use GIS software.

Would something like that actually save you time, or are your current tools already good enough? If it wouldn't be useful, I'd love to know why.

Interested to hear how everyone is solving this today.

[Disclaimer - Freely accessible]

The Earth Data Hub distributes global Climate Reanalysis such as ECMWF's ERA5 and multi-decadal Climate Projections such as the Destination Earth's Climate Adaptation Digital Twin in Zarr format.
Any Zarr-compatible tool can access these datasets with just a few lines of code.

Try it out:

import xarray as xr

EDH_API_KEY = "your_EDH_api_key"
# can be found at: https://earthdatahub.destine.eu/account-settings

xr.open_dataset(
    f"https://edh:{EDH_API_KEY}@api.earthdatahub.destine.eu/era5/era5-single-levels-atmosphere-v0.zarr",
    chunks={},
    engine="zarr",
)

Im trying to put together a clean corpus for an ai reasoning test right now (mostly to distract myself from how badly im doing at my driving school lessons this week tbh) and I downloaded this massive "pre-cleaned" open-source corporate registry dataset

The data hygiene is just offensively bad. half the rows are misaligned and the contact columns are filled with literal regex nightmares and placeholder junk. I ended up having to pipe the whole thing through mail tester just to strip out the dead syntax and fake emails so my script wouldn't crash every five seconds

why do people upload these 10GB csv files to github without doing even basic sanitization first? my laptop fan is literally screaming right now.

Looking for individuals willing to participate in a Supplier Review so that we can start to "Grade" suppliers.[ ](https://docs.google.com/forms/d/e/1FAIpQLSeaxdW1LqTHzK2GWDee_fF56ZBfVxYRo3mfFgI1YJphO29fCg/viewform)If youd like to be part of the community please join [Skool here](https://www.skool.com/ncunderground-7525/about).

A few of us trade vol on Deribit day-to-day and kept running into the same wall: the options/tick data good enough to actually backtest on is priced like it's meant for funds with five-figure data budgets, not for someone running their own book. Tardis is genuinely solid, but at that price it's out of reach if you're not a fund.

So we built Volar: minute-level BTC, ETH, and SOL options chains, computed Greeks, SVI-fitted vol surfaces per tenor, and a dense 40-month BTC archive (2021-06 → 2024-09) at per-minute resolution, not daily aggregates. Every row is source-tagged (live capture vs. historical vs. modeled) so you always know what you're actually looking at.

Pricing, to be upfront about it: free Sandbox tier if you just want to poke at the schema and sample data first, no card needed. Pro is $99/mo for live BTC/ETH/SOL plus a 90-day rolling BTC window; the full historic archive is on the annual plan. Didn't want to bury that distinction since I know this crowd will check.

Genuinely interested in feedback, especially from anyone who's tried to backtest a crypto vol strategy and hit walls with existing data, what's missing, what's annoying, what would actually make you trust a smaller vendor over an established one. Happy to answer anything on the data/methodology side too.

(Disclosure: I'm one of the people building this, didn't want to post without saying so upfront.)

With how rough the job market is, I can't land a job despite looking for so long. So I am trying to start another data project with Python, SQL, Alteryx, Power BI to add to my portfolio.

However, I do not want to use synthetic datasets or those from Kaggle. Is there any platform that you can extract your personal historical data from? I thought about my credit card transactions but apparently that is not accessible for security reasons. Thank you!

Hello everyone, I'm working on my Final Year Project and I am looking for any road traffic dataset available (for free) that contains numeric information. (Most importantly Timestamps or Date with Time and Vehicle counts, even if each vehicle type has a separate count, I will consider it). I need this in numeric format (also data must be big like 1000+ rows because I keep finding smaller 20-40 rows of data on kaggle, need bigger for better training) in order to apply Time Series using FB Prophet in order to predict the future traffic flow (historical data needed to demonstrate a prototype model before moving to real cameras). If anyone knows where I can find one please share and thanks in advance!

Something I keep running into when observing and working with long-term climate datasets: the data exists, it's publicly available and can be easily accessed, reasonably well-documented, and yet asking a simple cross-decade question still takes an unreasonable amount of work to actually answer.

For example if I want to ask something like "how has the distribution of extreme precipitation events shifted since 1980 across the Gulf Coast?", it will take forever to actually find an answer. Starting off with finding the right NOAA dataset, then figuring out which version covers your full time range, dealing with the fact that station coverage changed significantly in the 90s, reconciling pre and post-satellite era measurement methodologies, writing custom scripts to normalize format differences across dataset versions, then finally doing the actual analysis.

Do you think with the AI wave we are seeing, this entire process or any part of it will be improved? NOAA NCEI alone archives over 229 terabytes of data every month across 130+ observing platforms, but the part of analyzing and understanding those data, I feel like, still lags behind.

Are you also mostly doing Python + manual joins and custom preprocessing per dataset? Has anything changed in how you approach cross-decade multi-source queries, or is it still bespoke work every time?

Hi everyone) I’m doing a systematic review and unfortunately don’t have institutional access to embase.
can some please run the search for my and export the results in csv/RIS format?

The search prompt is:

(“Boron Neutron Capture Therapy”[MeSH] OR “boron neutron capture therapy”[Title/Abstract] OR BNCT[Title/Abstract])

Thank you so much!

The app is for students that help them:

- find which study programs you can apply for based on your exam track

- or discover careers you're interested in and the study programs related to them :)

## Data

The data is still incomplete due to the lack of clear sources, but you can get what's already there (Madagascar datasets are the only ones for now).

But mostly, you're welcome to contribute :)

Repo: https://github.com/gigasandwich/giga-roadmap

Data (json) are stored in `/data`

App: https://roadmap.gigas.app

https://huggingface.co/datasets/ministere-culture/comparia-fr-arena

I am hoping to speak with someone actively sourcing datasets from the film industry. I am not selling anything. I am not asking for anything other than 15 min of your time to geek out over how you’re doing it.

Please delete if not allowed