You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, our perf-cost experiment does not attempt to provide any cost estimations. We should provide this functionality, hardcode cloud costs per supported regions (or query them, if there's an API for it):
Hi @mcopik, following up on your email, I'd like to take this one. I went through the billing code and the provider backends first, so here's what I found and my plan.
Only AWS fills in the billing fields right now (billed_time, memory, gb_seconds in sebs/aws/aws.py). Azure and GCP get execution time but no billing data, and gb_seconds defaults to 0, so multiplying it by a price would make them look free. I'd leave the cost empty when there's no data.
The providers also don't bill compute the same way:
Provider
Compute is billed on
What SeBS has today
AWS
billed time x configured memory (GB-s), rate depends on region and x86/ARM
billed time and memory per invocation
Azure Consumption
GB-s using sampled memory, rounded up to 128 MB
execution time per invocation, memory is still a TODO
GCP gen1
GB-s + GHz-s
execution time per invocation, plus monitoring metrics
GCP gen2 / Cloud Run
vCPU-s + GiB-s per instance, and one instance can serve many requests at once
execution time per invocation, plus service-level metrics like billable_instance_time
So I think AWS can have a real cost per invocation, but the others probably need a total per experiment. For Azure, FunctionExecutionUnits in Azure Monitor looks like the right number, but it's for the whole app.
For the first PR I'd do AWS only:
In the perf-cost process step, work out compute + request cost per invocation from billed_time and memory, and add it to the processed JSON and result.csv.
Print a total per experiment, and how many invocations had no billing data, since the docs say some can be missing from the logs.
Keep prices in a JSON file in config/ by region and architecture with an "as of" date, filled from AWS's public price file.
Ignore the free tier and volume discounts, so it's the cost of the run and not your real bill. For example 968 ms at 128 MB on x86 in us-east-1 is 0.121 GB-s, about $0.0000022 with the request fee.
I'd leave out Azure, GCP, storage (the AWS Python wrapper doesn't count storage requests as far as I can see) and workflow state transitions (still in #303) for now.
Few questions:
Is a price file ok or would you rather query pricing APIs? I prefer the file because reprocessing the same results then gives the same cost, and GCP's API returns 403 without a key.
The HTTP trigger uses a Function URL by default, but API Gateway adds its own per-request fee. Ok to leave that out for now and just note it?
Could you share one processed AWS perf-cost result? I'd like to check my numbers against real output.
Also, docs/experiments.md still shows _gb_seconds: 123904, which is from the old billed_time * memory formula before #244. I'll calculate from billed_time and memory so older result files work too, and I can fix the docs example in the same PR if you want.
Activity
limonuser commentedon Nov 5, 2026
Hi @mcopik, following up on your email, I'd like to take this one. I went through the billing code and the provider backends first, so here's what I found and my plan.
Only AWS fills in the billing fields right now (
billed_time,memory,gb_secondsinsebs/aws/aws.py). Azure and GCP get execution time but no billing data, andgb_secondsdefaults to 0, so multiplying it by a price would make them look free. I'd leave the cost empty when there's no data.The providers also don't bill compute the same way:
billable_instance_timeSo I think AWS can have a real cost per invocation, but the others probably need a total per experiment. For Azure,
FunctionExecutionUnitsin Azure Monitor looks like the right number, but it's for the whole app.For the first PR I'd do AWS only:
perf-costprocess step, work out compute + request cost per invocation frombilled_timeandmemory, and add it to the processed JSON andresult.csv.config/by region and architecture with an "as of" date, filled from AWS's public price file.I'd leave out Azure, GCP, storage (the AWS Python wrapper doesn't count storage requests as far as I can see) and workflow state transitions (still in #303) for now.
Few questions:
perf-costresult? I'd like to check my numbers against real output.Also,
docs/experiments.mdstill shows_gb_seconds: 123904, which is from the oldbilled_time * memoryformula before #244. I'll calculate frombilled_timeandmemoryso older result files work too, and I can fix the docs example in the same PR if you want.