Stream Reddit Comments using Elixir, Pythonx and PRAW

Here's the thing about Reddit - it's simultaneously the best and worst place on the internet. You go there looking for legitimate discussion about Arsenal's latest transfer window (more on that later), and somehow end up three hours deep in a rabbit hole analysing which Game of Thrones actors actually hated the show. Oh, and let's not even talk about the constant battle of trying to browse /r/all while avoiding the NSFW content that seems to pop up at the most inconvenient moments - nothing quite like explaining to your coworker why there's suddenly adult content on your screen.

But here's what I love about Reddit: it's where real conversations happen. It's raw, unfiltered, and when you find the right communities, it's pure gold.

As an Arsenal fan, /r/gunners has been my hideout through the ups and downs of supporting this beautiful, frustrating club. And speaking of ups - yesterday's transfer news has actually given us something to be excited about for once!

The timing couldn't be better for building a Reddit comment streaming bot.

The Magic of Pythonx: Python Meets Elixir

Enter Pythonx - a game-changing library that embeds a Python interpreter directly into your Elixir application. Think about this for a moment: we can now leverage the entire Python ecosystem while staying in the warm embrace of Elixir's fault-tolerant, concurrent world.

This isn't just a neat party trick. This is revolutionary for those of us who chose Elixir for its incredible concurrency model, fault tolerance, and general awesomeness, but occasionally found ourselves staring enviously at Python's rich ecosystem of libraries. Want to use PRAW for Reddit API access? Done. Need scikit-learn for machine learning? Easy. Craving some pandas for data manipulation? You got it.

The beautiful thing about Pythonx is that it runs the Python interpreter in the same OS process as your Elixir application. No network calls, no serialization overhead, no distributed systems headaches. It's like having your cake and eating it too, except the cake is concurrent and fault-tolerant.

Why This Matters for Elixir Applications

Let's be honest - Elixir is fantastic, but sometimes we need functionality that just doesn't exist in our ecosystem yet. Before Pythonx, you had a few options:

• Port/System.cmd to shell out to Python scripts (clunky and slow)
• Write native implementations (time-consuming and potentially error-prone)
• Use HTTP APIs to Python services (network latency and complexity)
• Just... give up and use Python for the whole thing (sacrilege!)

None of these felt right. We chose Elixir for good reasons - its actor model, fault tolerance, and incredible performance characteristics for concurrent applications. Why should we abandon these benefits just because we need to parse some Reddit comments?

Pythonx changes everything. Suddenly, we can:

• Build real-time data pipelines using Python's data science libraries
• Integrate machine learning models directly into Phoenix applications
• Leverage specialized Python packages for domain-specific tasks
• Prototype faster by using existing Python solutions

Building a Reddit Comment Streamer

Let's dive into building something practical. I wanted to create a live comment stream for /r/gunners because, frankly, with all the transfer speculation happening right now, I need to stay on top of every rumor, every ITK (In The Know) update, and every meltdown in real-time.

Let's take a look using Livebook ❤️

The setup is surprisingly elegant:

Mix.install([
  {:kino, "~> 0.12.0"},
  {:pythonx, "~> 0.4.0"}
])

Pythonx.uv_init("""
[project]
name = "project"
version = "0.0.0"
requires-python = "==3.13.*"
dependencies = [
  "praw==7.7.1"
]
""")

What's happening here is pretty cool. We're using Pythonx's uv_init to set up a Python virtual environment with our dependencies. Behind the scenes, this creates an isolated Python environment that our Elixir process can interact with directly.

Then we initialize our Reddit connection:

{_result, globals} =
  Pythonx.eval("""
  import praw
  reddit = praw.Reddit(
        client_id="your_client_id",
        client_secret="your_client_secret", 
        user_agent="livebook",
    )
  subreddit = reddit.subreddit("gunners")
  """, %{})

The globals variable is key here - it maintains the Python execution context between calls. This means our reddit and subreddit objects persist and can be reused in subsequent Python evaluations.

The Real-Time Streaming Loop

Here's where things get interesting. We create a spawned process that continuously polls for new comments:

spawn(fn ->
  loop = fn loop, seen_ids ->
    {result, _} =
      Pythonx.eval("""
      [ (c.id, c.body[:100]) for c in subreddit.comments(limit=10) ]
      """, globals)

    new_comments =
      result
      |> Pythonx.decode()
      |> Enum.reject(fn {id, _} -> MapSet.member?(seen_ids, id) end)

    seen_ids = Enum.reduce(new_comments, seen_ids, fn {id, _}, acc -> 
      MapSet.put(acc, id) 
    end)

    if new_comments != [] do
      Kino.Frame.render(frame,
        Enum.map(new_comments, fn {_id, body} ->
          "* #{body}"
        end)
        |> Enum.join("\n")
        |> Kino.Markdown.new()
      )
    end

    Process.sleep(1000)
    loop.(loop, seen_ids)
  end

  loop.(loop, seen_ids)
end)

This is where Elixir really shines. We're using:

• Spawned processes for concurrency
• Tail-recursive functions for efficient looping
• MapSet for fast lookups of seen comment IDs
• Pattern matching for clean data handling

The Python code is doing the heavy lifting of interacting with Reddit's API through PRAW, while Elixir handles the application logic, state management, and real-time updates.

The GIL Reality Check ⚠️

Now, let's address the elephant in the room: Python's Global Interpreter Lock (GIL). The GIL prevents multiple threads from executing Python code simultaneously, which means if you're calling Pythonx from multiple Elixir processes, you won't get the concurrency you might expect.

For our Reddit streaming use case, this is actually fine. We're making I/O-bound API calls, and Python releases the GIL during I/O operations. Plus, PRAW handles rate limiting internally, so we wouldn't want true parallelism anyway - Reddit's API has strict rate limits.

But this is crucial to understand for other use cases. If you're doing CPU-intensive Python work, consider:

• Keeping Python operations in a single Elixir process
• Using libraries like NumPy that release the GIL for intensive operations
• Falling back to System.cmd/3 or Ports for truly parallel Python execution

What This Unlocks

The possibilities are genuinely exciting:

Real-time Analytics: Stream social media data through Python libraries, process it in Elixir's concurrent environment, and serve real-time dashboards through Phoenix LiveView.

Machine Learning Integration: Load trained models with scikit-learn or TensorFlow, make predictions in Python, and integrate the results seamlessly into Elixir applications.

Data Pipeline Hybrid: Use pandas and NumPy for complex data transformations, then leverage Elixir's GenServer and GenStage for reliable, fault-tolerant pipeline processing.

Rapid Prototyping: Quickly integrate existing Python solutions while building the foundation in Elixir, then gradually port critical paths to native Elixir as needed.

The Arsenal Connection

Why /r/gunners specifically? Well, yesterday's news about our potential signing has the entire subreddit buzzing, and as any Arsenal fan will tell you, transfer windows are emotional rollercoasters. Having a real-time stream of comments means I can catch every reaction, every rumor, and every tactical analysis as it happens.

Plus, let's be honest - Arsenal fans are some of the most creative and passionate commenters on Reddit. The memes alone make it worth monitoring. And after years of "Trust the Process" and watching our team evolve, there's finally some genuine excitement in the air.

Looking Forward

Pythonx represents something bigger than just Python integration - it's about breaking down artificial barriers between programming ecosystems. We no longer have to choose between Elixir's concurrency model and Python's library ecosystem. We can have both.

This opens up Elixir to entirely new domains. Data science applications that need real-time processing. Machine learning systems that require fault tolerance. API integrations that benefit from Python's mature libraries but need Elixir's reliability.

For indie hackers and solo founders, this is particularly powerful. You can leverage the massive Python ecosystem for rapid prototyping and specialized functionality, while building your core application logic in Elixir's robust, scalable environment.

The Reddit comment streamer is just the beginning. What will you build with Python's libraries and Elixir's superpowers combined?

---

If you're building with Elixir and want more content like this,I hope you found this post useful, subscribe to my Substack below for similar content and follow me on Twitter and Bluesky for more deep dives into the tools and techniques that make modern development faster and more enjoyable. And if you're an Arsenal fan... well, COYG! 🔴