Skip to content

Cost estimations #285

Open
Open

Description

@mcopik

Currently, our perf-cost experiment does not attempt to provide any cost estimations. We should provide this functionality, hardcode cloud costs per supported regions (or query them, if there's an API for it):

  • Function duration and memory costs
  • Function invocation costs
  • State transitions for workflows
  • Estimate storage accesses

Activity

  1. limonuser commented on Nov 5, 2026

    @limonuser

    Hi @mcopik, following up on your email, I'd like to take this one. I went through the billing code and the provider backends first, so here's what I found and my plan.

    Only AWS fills in the billing fields right now (billed_time, memory, gb_seconds in sebs/aws/aws.py). Azure and GCP get execution time but no billing data, and gb_seconds defaults to 0, so multiplying it by a price would make them look free. I'd leave the cost empty when there's no data.

    The providers also don't bill compute the same way:

    Provider Compute is billed on What SeBS has today
    AWS billed time x configured memory (GB-s), rate depends on region and x86/ARM billed time and memory per invocation
    Azure Consumption GB-s using sampled memory, rounded up to 128 MB execution time per invocation, memory is still a TODO
    GCP gen1 GB-s + GHz-s execution time per invocation, plus monitoring metrics
    GCP gen2 / Cloud Run vCPU-s + GiB-s per instance, and one instance can serve many requests at once execution time per invocation, plus service-level metrics like billable_instance_time

    So I think AWS can have a real cost per invocation, but the others probably need a total per experiment. For Azure, FunctionExecutionUnits in Azure Monitor looks like the right number, but it's for the whole app.

    For the first PR I'd do AWS only:

    • In the perf-cost process step, work out compute + request cost per invocation from billed_time and memory, and add it to the processed JSON and result.csv.
    • Print a total per experiment, and how many invocations had no billing data, since the docs say some can be missing from the logs.
    • Keep prices in a JSON file in config/ by region and architecture with an "as of" date, filled from AWS's public price file.
    • Ignore the free tier and volume discounts, so it's the cost of the run and not your real bill. For example 968 ms at 128 MB on x86 in us-east-1 is 0.121 GB-s, about $0.0000022 with the request fee.

    I'd leave out Azure, GCP, storage (the AWS Python wrapper doesn't count storage requests as far as I can see) and workflow state transitions (still in #303) for now.

    Few questions:

    1. Is a price file ok or would you rather query pricing APIs? I prefer the file because reprocessing the same results then gives the same cost, and GCP's API returns 403 without a key.
    2. The HTTP trigger uses a Function URL by default, but API Gateway adds its own per-request fee. Ok to leave that out for now and just note it?
    3. Could you share one processed AWS perf-cost result? I'd like to check my numbers against real output.

    Also, docs/experiments.md still shows _gb_seconds: 123904, which is from the old billed_time * memory formula before #244. I'll calculate from billed_time and memory so older result files work too, and I can fix the docs example in the same PR if you want.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions