John A De Goesramblings of a geekJekyll2023-05-19T07:54:52-06:00https://degoes.net/John A De Goeshttps://degoes.net/john@degoes.nethttps://degoes.net/articles/splendid-scala-journey2023-05-19T00:00:00-06:002023-05-19T00:00:00-06:00John A De Goeshttps://degoes.netjohn@degoes.net<p>I have been a Scala developer for 15 long years, which makes me one of the oldest Scala programmers around.</p>
<p>My journey began when, as VP of Engineering at a San Francisco startup, I needed to choose a programming language for our new ad-tech platform.</p>
<p>Without hesitation, I chose Scala, because I had a delightful experience dabbling with functional programming in Java, where I had spent the previous 10 years mastering object-oriented programming.</p>
<p>Unfamiliar with Scala, I delved into Odersky’s book, devouring it cover-to-cover during my weekly commutes between Denver and San Francisco</p>
<p>The more I learned about Scala—its lambdas, futures, actors, generics, powerful collection library, and the expression-orientation of the language—the more enthralled I became. I felt like a kid in a candy store, delighting in each new discovery!</p>
<p>In the 15 years since that fateful decision to build an ad-tech platform in Scala, I have built 3 more companies on Scala, spent more than 20 million on Scala engineers, and built the <a href="https://zio.dev">large, thriving Scala ZIO ecosystem</a>, which has gone on to inspire ports in TypeScript, Kotlin, F#, and other programming languages.</p>
<p>Through countless talks, articles, 1-on-1 mentorship, and educational workshops, I have become a vocal Scala evangelist and champion, tirelessly advocating for Scala within industry. Thanks to my efforts, tens of thousands of developers around the world have learned Scala and leveled-up their skills and knowledge.</p>
<p>I witnessed the rise of Spark and Twitter Scala, the evolution and transformation of Lightbend (formerly Typesafe), and so much more.</p>
<p>Yet, despite these successes and my deep-rooted passion for Scala, I find recent trends drawing me towards new horizons.</p>
<h2 id="state-of-scala">State of Scala</h2>
<p>Ever since 2017, I have observed a decline in Scala’s adoption within industry—a trend I <a href="https://www.youtube.com/watch?v=v8IQ-X2HkGE">first spoke about in 2018</a>. I’ve discussed Scala’s adoption challenges more recently in a <a href="/articles/scala-resurrection">post</a> and <a href="https://twitter.com/jdegoes/status/1652940041524486145">several</a> <a href="https://twitter.com/jdegoes/status/1656566825356754945">Twitter</a> <a href="https://twitter.com/jdegoes/status/1658801254196232195">threads</a>.</p>
<p>The transition from Scala 2 to Scala 3 has proven extremely challenging for the industry. Scala 3 is essentially a new language, and upgrading large projects that rely on unmaintained libraries or macros is not a simple task. The majority of commercial users are still on Scala 2 and lack significant incentives to upgrade to Scala 3. Confronted with investing months into upgrades that offer no new business value, many companies are looking to alternative languages that better meet their needs.</p>
<p>Further complicating matters, Scala 3 has had numerous bugs, including compiler crashers, and has seen rapid evolution. It has introduced new features that the industry isn’t demanding and still lacks industry-grade tooling. These issues risk <a href="https://www.reddit.com/r/scala/comments/12zhsoz/is_scala_losing_ground_in_your_organization/">burning out organizations</a>, developers, and OSS contributors.</p>
<p>Despite my ongoing efforts to champion and evangelize Scala, watching the decline in adoption year after year has left me with a mix of frustration and sadness. Frustration, because I wanted to address the core challenges hindering its adoption; sadness, because I longed for Scala to become mainstream.</p>
<p>Initially, I tried to influence change by becoming a commercial sponsor of the official Scala Center and joining the advisory board. When that didn’t happen, I started speaking candidly about Scala’s challenges and sharing my thoughts on how to improve adoption.</p>
<p>Today, I can say without hesitation that I have done literally everything that I know how to do. I have said everything I can possibly say, even at significant personal cost, in order to maximize the chance of Scala’s growth.</p>
<p>Letting go of something you deeply love is very difficult, especially if you believe (rightly or wrongly) that you understand the problems at hand and how to address them. The struggle has been genuine and often painful for me.</p>
<p>Yet, acknowledging that I’ve given my all and there’s nothing more to be done has led to a gradual fading of my frustration and disappointment.</p>
<p>Today, I’m neither frustrated nor disappointed. Instead, I feel a growing sense of acceptance and inner peace.</p>
<h2 id="the-future-of-scala">The Future of Scala</h2>
<p>At this point, I do not know if Scala will grow within industry, or if it will become a boutique programming language that a few lucky companies wield for its incredible expressive power and strong type-safety—paying for it, as they must, with the taxes demanded by all boutique programming languages.</p>
<p>Looking at Clojure, I appreciate that even though Clojure is a boutique programming language, it is still a beast of a programming language, which gives its commercial users a competitive edge, primarily through the senior talent it attracts, who is able to effectively channel its powers.</p>
<p>If the future of Scala is similar to Clojure, then I fully accept it. This future was not my preferred choice for the language, but it’s still a win for some savvy companies and talented developers, and that’s enough for me.</p>
<p>Moreover, there’s another, more positive way to look at Scala 3 adoption. Scala 3 is essentially a new programming language. Viewed in this light, its adoption and tooling are actually quite impressive!</p>
<p>Whatever happens to Scala, I know that it is not going to disappear. There will always be companies using Scala. Whether the market is large or boutique, I accept the fate of Scala, without reservation or complaint.</p>
<p>Que sera, sera.</p>
<h2 id="my-future-with-scala">My Future with Scala</h2>
<p>In light of the shifting market, as well as my long tenure in the Scala community, I am officially stepping down as Scala evangelist and champion.</p>
<p>I no longer consider myself a dedicated Scala developer, and I will no longer lobby for any Scala improvements, whether at the language or organizational level. I will not involve myself in any discussions regarding the language, standard library, or tooling (though I may promote or sponsor tooling). After 15 long years, it’s time for me to make room for the next generation of highly-capable Scala champions.</p>
<p>In response to these changes, I am diversifying my educational content, open source contributions, and mentorship, expanding into other programming languages. I’m especially interested in new languages that, like Scala, empower us with expressive power and support us in writing correct code.</p>
<p>Let me be clear: I still have a deep affection for Scala, both the 2.x and 3.x dialects. The language has been my trusty steed, and together we have fought and won many battles in industry.</p>
<p>However, while the past 15 years of my life have been dedicated almost exclusively to Scala, the next 15 years (should I be so fortunate) will expand beyond Scala and to new technologies.</p>
<p>I am grateful for having the chance to grow as a software engineer using such a powerful, type-safe programming language as Scala. Simultaneously, I am excited about the new journey ahead and the opportunities it presents for personal and professional growth.</p>
<p>Currently, I’m thoroughly enjoying writing some Rust (alongside Scala), and I look forward to teaching a <a href="https://www.eventbrite.com/e/learn-the-rust-programming-language-by-john-a-de-goes-tickets-630610863067">Rust workshop</a> for anyone interested in learning this powerful, type-safe, and remarkably efficient programming language.</p>
<h2 id="the-future-of-zio">The Future of ZIO</h2>
<p>As I am the founder of the ZIO ecosystem, it’s reasonable to ask: <em>How will this new journey affect the ZIO ecosystem?</em></p>
<p>To begin with, it’s crucial to recognize that even though I currently serve as the BDFL of the ZIO organization, I am only <em>one</em> contributor among hundreds. The significant contributions made by other skillful and productive developers have long surpassed my own.</p>
<p>In the early days of ZIO, yes, it was primarily <em>me</em>, working tirelessly during nights and weekends. Today, however, ZIO comprises hundreds of contributors from dozens of companies, including multiple Scala consultancies. ZIO has grown to be far larger than any one person, myself included.</p>
<p>These days, I am not even involved in the majority of ZIO projects!</p>
<p>Next, despite my upcoming ventures beyond Scala, I have no plans to stop contributing to the ZIO ecosystem, both directly and through mentorship of new contributors. Currently, I am heavily involved with <em>ZIO HTTP</em>, which I believe is a game-changer for ZIO users. I will continue pushing the ZIO ecosystem forward and supporting those who share this goal.</p>
<p>As for my own company, <a href="https://ziverge.com">Ziverge</a>, we continue to offer commercial support contracts for ZIO, as does <a href="https://scalac.io">Scalac</a>. Both companies provide commercial training, feature acceleration, and priority fixes across the whole ZIO ecosystem. With ZIO’s monthly downloads doubling to 8 million in the past year, growth remains strong, and this momentum will certainly drive further improvements.</p>
<p>Regardless of whether Scala’s adoption surpasses 2017 levels or settles into a small but highly skilled community like Clojure, functional Scala developers will always need a technology stack to rapidly build cloud-native applications. It is my goal for ZIO to remain one of the top choices!</p>
<h2 id="a-moment-of-reflection">A Moment of Reflection</h2>
<p>Looking back at my 15-year journey with Scala, I recall mostly good times. However, I have also navigated through rough patches, which have provided me ample opportunities for introspection.</p>
<p>I’ve had <a href="https://www.linkedin.com/pulse/dark-side-speaking-tech-conferences-john-de-goes/">two talks of mine cancelled</a>, I was shadow-banned from ScalaDays, and I endured years of <a href="/articles/travis-brown-abuser">cyber-stalking, defamation, and harassment</a>. Over the years, personal attacks, false accusations, and attempts to harm my professional relationships have become sadly familiar occurrences. Some have refused to provide proper attribution, a basic courtesy in our field.</p>
<p>Sometimes, the Scala community can feel like a magnet for drama, and I acknowledge that I have unintentionally contributed to this atmosphere at times, through either my choice of topics or the language I used.</p>
<p>When we suffer wrongs, we naturally experience hurt and anger, and we yearn to fight back. Despite my best efforts to manage these feelings, I acknowledge that I occasionally struggle with them, and these emotions might bleed into my words.</p>
<p>However, I remain steadfast in my belief in and practice of forgiveness. This is not merely because releasing negative emotions is healthy for us, but also because we ourselves need forgiveness. To be forgiven, we must forgive.</p>
<p>Forgiveness does not imply discarding the boundaries that protect us from further harm. Nor does it require trusting someone who has broken our trust. Forgiveness entails releasing negative emotions, allowing them to flow through us rather than get lodged within us, and ceasing our attempts to retaliate.</p>
<p>To those who have, in my eyes, wronged me—whether intentionally or not—I unconditionally forgive you, with no expectations in return. If I have said or done anything to hurt you, even unintentionally, I am open to a personal conversation to own my part.</p>
<p><strong>To Odersky</strong>:</p>
<p>I extend my heartfelt gratitude to you for creating such a powerful, type-safe language that has managed to engage me deeply for a span of 15 years.</p>
<p>You paved the way for object-oriented programmers like me to explore the magical new realm of functional programming. This shift has drastically altered my perspective and elevated my skills.</p>
<p>We do not always agree. In fact, our disagreements could probably fill a small book! But, regardless of our differences, I am profoundly thankful that you chose to create and share the Scala programming language with the world. Its influence on other ecosystems ensures Scala’s legacy will continue for decades.</p>
<p>Please bear in mind that when developers appear somewhat harsh in their feedback, it’s often driven by a deep-seated love for Scala. They might feel frustrated because they can’t shape the language as they wish. Every technology has its frustrated lovers, who should resist the temptation to become haters.</p>
<p>I have come to terms with whatever direction Scala takes from here. I understand the benefits of both language experimentation and stabilization. While I might place greater weight on one over the other, I recognize that each holds value for different audiences and for diverse reasons.</p>
<p>I wish you, and the entire team at EPFL, the very best for the future!</p>
<p><strong>To the Scala Center</strong>:</p>
<p>I have not expressed my gratitude nearly as much as I should have, but I want to acknowledge the immense contributions you make to the Scala community. Without the Scala Center, we would be in a far less favorable position.</p>
<p>Over the past few years, your mission to expand Scala’s adoption has been my own personal mission. Although I may have strong opinions (perhaps unreasonably so!) on how to achieve this goal, I fully recognize that it is, ultimately, <em>your responsibility</em> to decide how to advance your mission.</p>
<p>Even though we have had disagreements on focus and organization at times, I wholeheartedly want you to succeed in your mission. With the many talented engineers in your organization, I am confident you possess the necessary engineering prowess to tackle some of Scala’s challenges.</p>
<p>I extend my best wishes for your success, and I stand prepared to personally contribute financially to your organization should you so desire.</p>
<p><strong>To SoftwareMill</strong>:</p>
<p>I want to express my deep appreciation for all that you contribute to the open-source community. Your contributions to the Scala ecosystem have been invaluable, providing enormous value to Scala developers.</p>
<p>Rather than dwelling on recent incidents, I will strive to remember the numerous good times we shared.</p>
<p>My heartfelt congratulations on your merger, and I sincerely hope that both SoftwareMill and VirtusLab achieve great success, not only within the realm of Scala but beyond it as well.</p>
<p><strong>To Typelevel</strong>:</p>
<p>I want to take this opportunity to appreciate the significant influence you’ve had on the advancement of pure functional programming in Scala.</p>
<p>Your tireless effort has led to the development of a rich ecosystem of purely functional software, going far beyond Scalaz, and providing substantial value to the industry.</p>
<p>We may always disagree over <code class="language-plaintext highlighter-rouge">F[_]</code>, type-class-first development, the names of type classes and methods, and occasionally, concurrency, interruption, and back-pressure semantics.</p>
<p>But even if I have spent many words devoted to our differences, ZIO and Typelevel are more similar than different.</p>
<p>We both have seen and believe in the power of pure functional programming to solve real problems. We both believe in the power of programs-as-values.</p>
<p>I miss the times when we would grab beers in some bar in San Francisco and complain about <code class="language-plaintext highlighter-rouge">PartialFunction[Any, Unit]</code>. What happened to those days?</p>
<p>Well, a lot happened, and if I am being honest with myself, we each contributed to the outcome. For my part, I wish I had argued for improvements in a more skillful way that made everyone feel good about the countless, thankless hours they poured into open source.</p>
<p>We always agreed more than we disagreed, and I know I appreciated so much about the early design of Cats, FS2, and IO, but maybe I should have said this early and often to balance out my technical criticism.</p>
<p>I probably should have dialed down my marketing for ZIO, more diligently pointed out the usual disclaimers with micro-benchmarks, and more frequently complimented you on the innovation occurring in your ecosystem (polling IO? Crazy cool! Smithy4s? Damn!).</p>
<p>I wonder if, through conversations that could only ever happen in person, we had found constructive, and uplifting ways of achieving our community-oriented goals. Could I have addressed your concerns, and could you have addressed and dealt with the negative experiences of my friends in the Scala community?</p>
<p>I don’t know if we would have found common ground, but we probably should have tried to have those conversations, face-to-face.</p>
<p>In any case, even though it’s beyond my power to make it happen at this point, I think ZIO and Typelevel working together could help Scala, and perhaps this can happen through the new generation of Scala open source contributors, who don’t have our baggage, and just want to see functional Scala succeed.</p>
<p>Whether that collaboration involves consolidating libraries that have a similar purpose, or working together on a <em>Better For</em>, the functional Scala community is strongest united on the side of <em>every</em> functional Scala developer.</p>
<p>Regardless of what happens in the future, I wish all of you the best of luck.</p>
<p><strong>To the ZIO community</strong>:</p>
<p>I am so happy and proud to be part of the ZIO open source ecosystem.</p>
<p>I love the fact that all of you are constantly challenging existing patterns and practices in functional programming, producing software that is powerful, practical, functional, accessible, and joyful.</p>
<p>I love the culture of the ZIO community. Not only are you talented, but you are open to work with everyone, non-judgmental, positive and upbeat (something I’ve failed at recently!), and eager to welcome and mentor all.</p>
<p>I’ve been completely blown away by the powerful, elegant libraries that you have created (ZIO Query, ZIO HTTP, Shardcake, Caliban, ZIO ElasticSearch, etc.), and the many new contributors that you have cultivated around these libraries.</p>
<p>Over the past couple of years, ZIO developers and users have been my primary motivation to invest into Scala. Even as I begin a new journey, expanding my horizons beyond Scala, the ZIO community remains the shining light that will keep me hooked on contributing to Scala open source.</p>
<p>I am eternally grateful to all of you for your kindness and support, for the amazing libraries that you create, and for your feedback using ZIO libraries, which help us make our open source libraries even better.</p>
<p><em>“ZIO soon,”</em> as they say, to all my friends in the ZIO community! We need to finish that Spring killer, and we’re getting awfully close!</p>
<h2 id="summary">Summary</h2>
<p>As one of the more seasoned Scala developers in the community, my 15-year journey with Scala has been an enriching experience. I’ve built four companies, employed numerous Scala developers, nurtured a flourishing open source ecosystem, and advocated tirelessly for Scala as a commercially viable functional programming language.</p>
<p>Despite my best efforts, industry adoption of Scala has faced certain challenges. After much deliberation, struggle, and public discourse, I’ve come to deeply accept this reality. Scala may well follow a path similar to that of Clojure. This wasn’t my initial vision for the language, but it’s still a positive outcome.</p>
<p>In recognizing this shift, I am officially stepping down from my role as a Scala evangelist and champion, and am warmly passing the baton to the next generation—those brilliant young minds tirelessly working on Scala’s compiler, tooling, and open source ecosystem. This new generation, with its fierce spirit and boundless energy, is surely up for the challenge.</p>
<p>While my love for Scala is unchanged, and I will continue to happily write Scala, my journey will take me to new places, and on new adventures.</p>
<p>The ZIO ecosystem will continue to flourish under the diligent stewardship of countless talented and committed individuals. Commercial users will still have access to support, training, feature acceleration, and priority fixes from two reputable consultancies. I will maintain my involvement by contributing to and guiding core projects.</p>
<p>As I embark on an exciting new journey, I reflect with profound gratitude on the extraordinary saga that has been my very splendid Scala journey.</p>
<p>Consummatum est.</p>
<p><a href="https://degoes.net/articles/splendid-scala-journey">A Splendid Scala Journey</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on May 19, 2023.</p>https://degoes.net/articles/new-scala-build-tool2023-04-11T00:00:00-06:002023-04-11T00:00:00-06:00John A De Goeshttps://degoes.netjohn@degoes.net<p>I previously <a href="https://degoes.net/articles/scala-resurrection">made a case</a> that Scala needs coordinated and radical changes in order to remain relevant to industry.</p>
<p>On the tooling front, the two areas where Scala needs the most help are IDEs and build tools.</p>
<p>In this post, I’d like to outline what I believe are the three separate paths to a best-in-class build tool, and share my thoughts on each strategy.</p>
<h2 id="path-1-ditch--run">Path 1: Ditch & Run</h2>
<p>The first path to having a good build tool for Scala is, ironically, to ditch all of our Scala build tools! Rather than using a Scala build tool, we embrace and extend an existing multi-language build tool, such as Gradle, Pants, or Bazel.</p>
<p>In order for this strategy to be successful, three ingredients are necessary:</p>
<ol>
<li>We collectively decide upon a multi-language build tool, and we migrate our projects to it.</li>
<li>We identify weak areas in the built tool’s support for Scala, and we contribute to smooth them over, with particular attention to new developer onboarding experience (required for the top of the funnel) and complex build requirements (required for the bottom of the funnel, where the very few large commercially relevant projects exist).</li>
<li>We engage in evangelism and education to teach Scala developers how to use this multi-language build tool, deprecating Scala build tools.</li>
</ol>
<p>This strategy was successfully adopted by Kotlin, which largely piggybacks on Gradle (most Kotlin projects, including Kotlin itself, are built using Gradle).</p>
<p>Personally, I do not think Bazel is appropriate, because of the extreme difficulty making Bazel accessible to top-of-the-funnel. The other two obvious choices are Gradle and Pants.</p>
<p>Pants V2 is written partially in Rust and is quite fast. Gradle is written in Groovy and Java, and inherits slow JVM startup time and large memory usage, though attempts to workaround the former with build daemons (like SBT and other Scala build tools).</p>
<p>The support for Scala in these build tools is immature (Gradle is <a href="https://github.com/gradle/gradle/tree/master/subprojects/scala/src/main/java/org/gradle/api">here</a> and Pants is <a href="https://github.com/pantsbuild/pants/tree/main/src/python/pants/backend/scala">here</a>), but the code base is relatively small and manageable, and should be straightforward to bring them up to speed with Scala build tools.</p>
<p>The main advantage of embracing an existing multi-language build tool is that the steps involved in building a software application are largely common across different languages. So rather than re-invent what all other build tools invent, we can just leverage existing machinery, and put our limited resources into polishing and extending the Scala plug-ins in these build tools.</p>
<p>Essentially, it’s like getting a huge workforce for free to build and maintain every part of the build tool except the Scala-specific parts, which includes not only the core build machinery (watching files, downloading dependencies, caching artifacts, etc.) but also a massive ecosystem of plug-ins that add features like publishing, packaging, deployment, and so forth.</p>
<p>The main disadvantages of adopting an existing multi-language build tool are as follows:</p>
<ol>
<li>Multi-language build tools don’t know anything about Scala, so they can’t tailor their user-interfaces and base feature sets specifically to Scala. Obviously, however, the language-specific plug-ins to these build tools can be built for Scala developers, so this limitation only affects the base feature set of the build tools.</li>
<li>Multi-language build tools are not written in Scala. To contribute to Gradle, you should know Java and Groovy. To contribute to Pants, you should know Python and maybe Rust. There is a much smaller pool of Scala developers who are able and willing to contribute in these languages.</li>
</ol>
<p>It’s worth pointing out that many large projects (including, for example, Twitter’s own code base) have long ago ditched Scala build tools, because their needs (especially for performance) grew more sophisticated than what our own build tools offer.</p>
<h2 id="path-2-double-down">Path 2: Double-Down</h2>
<p>The second path to having a good build tool for Scala is to double-down on existing build tools, and try to address their weaknesses.</p>
<p>The dominant Scala build tool is SBT, which has <a href="https://degoes.net/articles/scala-resurrection">serious drawbacks</a>. So realistically, we are talking about doubling down on SBT.</p>
<p>The main problem with this path is that fixing everything that’s wrong with SBT would completely break backward compatibility. Which means, in essence, it would be a Python 2/3 or Scala 2/3 catastrophe, where users would have to develop new build files or try to “port” their build files to what is essentially a new build tool with a new ecosystem.</p>
<p>If you are going to break backward compatibility, and force a painful and time-consuming transition, then effectively, this path is the same as building a new build tool.</p>
<h2 id="path-3-new-scala-build-tool">Path 3: New Scala Build Tool</h2>
<p>The final path to having a good build tool for Scala is to <em>create</em> a new Scala build tool, either from scratch or by forking an existing Scala build tool without the constraint of backward compatibility.</p>
<p>I think it quite likely any new Scala build tool will end in ruin. There are dozens of dead Scala build tools, very few of which ever managed to gain any traction.</p>
<p>There are a few reasons for this inglorious end for new Scala build tools:</p>
<ol>
<li><strong>Hobbies</strong>. Scala build tools are written by hobbyists with an itch to scratch, not by commercial developers who are being paid by their employers to solve real problems. Hobbyists frequently abandon their work, and just as frequently only solve the “interesting” part of a problem, not the “boring” part. For build tools, however, solving the boring part is extremely important to adoption.</li>
<li><strong>Two-Sided Market</strong>. There is no successful build tool that is successful by itself: they all have large ecosystems that add incredibly useful features to the build, package, and deployment process. For example, SBT has a large network of plug-ins that solve common problems. Without such an ecosystem, a build tool has trouble attracting users. Yet, without users, developers aren’t motivated to build out an ecosystem. It’s a chicken-egg problem.</li>
<li><strong>Inexperience</strong>. Most developers have no experience in building successful products, and every open source project (including tools) is a product (or a mini-startup, if you will, unfunded and broke!). Building successful products requires knowledge and discipline: knowledge of what to build (features) and how to build it (usability), how to talk about it in a way that drives growth (marketing), what features to focus on at what stage (product triage), and so much more.</li>
</ol>
<p>Without taking an opinion on whether or not investing in a new Scala build tool is a great idea, let me outline my thoughts on what such a build tool should focus on to succeed. Rather than focus on obvious features, I’ll cover some of the less obvious but crucial features.</p>
<p>In no particular order:</p>
<h3 id="build-as-data">Build-as-Data</h3>
<p>A build file should be pure data, not code, a mistake that SBT and other build tools have replicated. The idea of expressing a build in a general-purpose programming language is attractive to generalists, but is a massive drawback to tooling, which wants to be able to read and write build files (this is especially important for IDEs).</p>
<p>In addition, build tools powered by programming languages trade off usability for expressive power that is utilized by almost no one, and which can be obtained in other ways (plug-ins). A user creating a new build file does not want to start from an empty Scala file, having no idea what to type or why. They want a few obvious and build parameters they can set. They want a Scala build-specific “language” that’s so highly constrained, it’s obvious what to write.</p>
<p>It’s important that the data be in a standardized format, which can be written and read by tooling. Moreover, unlike Bazel, I would say it’s important the build file format be high-level, and suitable for humans to create and maintain. Finally, it’s important the format be textual, so that developers can work on independent changes to the build file, and merge them in using standard Git-based workflows and tooling.</p>
<p>A proprietary binary format for the build file (or even a textual format that doesn’t merge well using version control) would be a catastrophic mistake, even if advanced front-end tooling existed to create and maintain such build data.</p>
<h3 id="all-in-on-scala">All-in on Scala</h3>
<p>Let’s face it: many programmers like to abstract. They like to build platforms, not products. They like the general, not the specific. This is doubly-true for developers who are attracted to Scala, because one of the main reasons to use Scala is its immense power of abstraction.</p>
<p>Making a Scala build tool that is so general-purpose it can build any project in any language is a gigantic mistake. Because there are already best-in-class multi-language build tools, which are effectively impossible to compete with on hobbyist resources.</p>
<p>It’s even a mistake to focus the build tool on “generic Scala”, without deeply integrating into the existing quirks and features of the Scala languages and compilers. SBT, despite being a Scala build tool, knows almost nothing about Scala, forcing developers to hard-code aspects of the Scala compiler and language into their build files, repeated endlessly across all projects.</p>
<p>A Scala build tool should have first-class support for the following Scala-specific features:</p>
<ul>
<li><strong>Platform</strong>. Scala already supports three platforms, including JVM, JS, and native (via LLVM). In the future, Scala may support more platforms. A Scala build tool should allow developers to specify which platforms a project supports, including compiling overlapping but slightly different code bases for each platform. The build tool should also know that when targeting the JVM, you can choose to target one or more versions of the JDK, and that some versions of the JDK might require new or different source files (for example, supporting Loom in JDK 19 or later).</li>
<li><strong>Versions</strong>. Scala exists in multiple flavors, which are incompatible with each other, and all important to support: 2.12, 2.13, and the radically different 3.x line, with potentially more versions down the road. A Scala build tool should allow developers to specify which of these versions they support, including compiling overlapping but slightly different code bases for each platform (for example, one file may compile under Scala 2.12, but another one might be required for both Scala 2.13 and Scala 3).</li>
<li><strong>Warnings</strong>. A Scala build tool should know about and standardize warnings across different versions of Scala. Warnings are important to configure for all major projects, but some Scala versions only support some warnings, and right now developers have to configure them as error-prone strings without any guidance from the build tool.</li>
<li><strong>Experimental Features</strong>. A Scala build tool should know about and standardize experimental features, which can be turned on and off in different versions of Scala.</li>
</ul>
<h3 id="first-class-jvm">First-Class JVM</h3>
<p>The JVM is still the only commercial-grade backend platform for Scala. As a result, any new Scala build tool must have strong support for the JVM.</p>
<p>With SBT, a common experience is exhausting all heap or metaspace while compiling. Diagnosing these problems takes expertise and knowledge of the JVM. Fixing them requires arcane and mostly undocumented knowledge, passed from developer-to-developer by copy/paste and lore.</p>
<p>A Scala build tool should, for itself, as well as the JVM target, have explicit knowledge and support for common JVM parameters, including heap size, metaspace size, stack size, and so forth. It should be a simple per-project setting to choose how much space to allocate to compiling or testing a project.</p>
<h3 id="batteries-included">Batteries Included</h3>
<p>A new Scala build tool must resist the temptation to push all functionality into ecosystem projects, which may or may not exist. Rather, a new Scala build tool must directly tackle the two-sided market problem by baking in the features common across all Scala builds: features that are so wholly indispensible, if your build tool doesn’t have them, it won’t get adoption.</p>
<p>These features include:</p>
<ol>
<li>Starting a REPL for a given project (and platform/version)</li>
<li>Testing a project, with integrations for common Scala test frameworks</li>
<li>Packaging a project, including generating Scaladoc</li>
<li>Publishing a project to a repository</li>
<li>Running main functions</li>
<li>Producing containers</li>
</ol>
<p>Testing, packing, and publishing should be CI-friendly, with the realization that the CI tool (e.g. Github Workflows, Jenkins, etc.) will provide authentication details for wholly automated workflows.</p>
<p>These features, some of which are trivial, while two are more involved (publishing packages and building containers) are a compelling feature set that is sufficient to drive adoption, assuming the build tool is in reasonable shape otherwise.</p>
<h3 id="performance">Performance</h3>
<p>A new Scala build tool should be committed to performance. Especially as performance is the main reason companies abandon Scala build tools.</p>
<p>A fast build is a delight, and if Scala has its own build tool, it should focus on delighting developers.</p>
<h3 id="plug-ins">Plug-ins</h3>
<p>A new Scala build tool should support plug-ins. Developers universally push back against plug-ins, because designing a plug-in architecture is complex, and maintaining it is expensive.</p>
<p>However, there is no possible way SBT could have become the dominant build tool in Scala without plug-ins. Plug-ins allow ordinary developers to reuse the build and deployment features common across large chunks of the ecosystem, thus helping them rapidly assemble and tweak their builds to their specific needs, no matter how complex.</p>
<p>It might be possible to support plug-ins as ordinary command-line processes, which are fed details from the build process. However, care should be taken with this approach, because if a Scala developer builds a JVM-based plug-in, then startup times could be prohibitive (there’s nothing to say, however, that plug-ins should be developed in Scala, even though it will be the first choice for many).</p>
<p>Any new Scala build tool that rejects plug-ins on grounds they are “too hard” will fail with the largest and most commercially relevant projects (even though it could obtain some traction for toy / hobbyist projects, which have unsophisticated needs met easily without plug-ins).</p>
<h3 id="bonus-points">Bonus Points</h3>
<p>A new Scala build tool should, ideally, be geared at <em>eventually</em> solving two other concerns:</p>
<ol>
<li><strong>Package Management</strong>. Publishing of packages on Scala-specific repository. For too long, Maven has been abused for publishing Scala packages. But with Scala going cross-platform, and having completely different compatibility considerations, a Scala-specific packaging solution could be a far better fit. In addition, Maven is old-school and it’s difficult to host or find a free host to publish new libraries, and this discourages new Scala developers who are also new to the JVM.</li>
<li><strong>Language-Server Protocol</strong>. A build tool already has to interface with the Scala compiler, and has intimate knowledge of the build settings. An LSP server could be directly integrated into a build tool, improving performance, simplifying architecture, reducing points of failure, reducing code size, and making maintenance easier.</li>
</ol>
<p>These concerns don’t have to be tackled right away, but they represent the sort of bold new thinking that’s required to effectively utilize limited resources to produce a best-in-class experience for Scala developers.</p>
<h2 id="summary">Summary</h2>
<p>In this blog post, I discussed the three potential paths to creating a best-in-class build tool for Scala: ditching Scala build tools in favor of multi-language build tools, doubling down on existing build tools such as SBT, or creating a new Scala build tool from scratch.</p>
<p>Of these paths, I believe that only the first and third are viable. Fixing what’s wrong with SBT would require changes of such magnitude, it’s effectively a new build tool, anyway.</p>
<p>If we end up creating a new Scala build tool (from an existing or new code base), I believe that it must focus on build-as-data, providing first-class support for Scala-specific features, embracing the JVM, offering batteries-included functionality, prioritizing performance, supporting plug-ins, and eventually solving packaging and language-server protocol concerns.</p>
<p>In the <a href="https://github.com/zio/">ZIO ecosystem</a>, we are hungry for a new build tool, and at <a href="https://ziverge.com">Ziverge</a>, we have clients who are frustrated with existing Scala build solutions.</p>
<p>Even though I have no intention to directly solve Scala’s build tool problem, I will be keeping an eye on developments in the space, and if I see significant investment into Pants or Gradle, or if I see a new Scala build tool that is getting the right foundation, then I will not hesitate to support these investments, including migrating ZIO ecosystem projects, sponsoring development of features, and publicly endorsing and evangelizing such solutions.</p>
<p><a href="https://degoes.net/articles/new-scala-build-tool">New Scala Build Tool</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on April 11, 2023.</p>https://degoes.net/articles/scala-resurrection2023-01-24T00:00:00-07:002023-01-24T00:00:00-07:00John A De Goeshttps://degoes.netjohn@degoes.net<p>As a long-time user and one of the earliest evangelists of the Scala programming language, and as CEO of a <a href="https://ziverge.com">tech company</a> that has <a href="https://zio.dev">heavily invested</a> in Scala open source, I’ve been <a href="https://www.youtube.com/watch?v=v8IQ-X2HkGE">acutely interested</a> in growing Scala adoption within industry.</p>
<p>Scala 3 has the most powerful static type system and metaprogramming facilities in any major programming language. The language’s strength in designing robust internal domain-specific languages like Spark has opened new doors into <a href="https://zio.dev">concurrent programming</a>, <a href="https://github.com/zio/zio-flow">distributed programming</a>, <a href="https://github.com/zio/zio-quill">database access</a>, and many other domains.</p>
<p>Yet, not all is sunshine and roses in the world of Scala:</p>
<ul>
<li>The Scala 3 revision of the language has proven every bit as challenging as I <a href="https://www.youtube.com/watch?v=v8IQ-X2HkGE">anticipated</a> it would be.</li>
<li>The tooling around Scala is worse now than in the 2.x days, and poor compared to even much newer languages like Go, Kotlin, and Rust.</li>
<li>Critical platform and tooling projects in the Scala ecosystem remain underfunded and underdeveloped.</li>
<li>Scala has completely intertwined its fate with the JVM, which is under constant threat from more modern alternatives.</li>
<li>Lightbend has effectively close sourced the entire Akka ecosystem.</li>
</ul>
<p>These factors, combined with the scarcity and high cost of senior Scala talent, have pushed many companies to other languages like Go, Kotlin, Java, and Rust, depending on their use cases.</p>
<p>While there are too many commercial users of Scala for the language to “disappear”, it is not clear that the Scala programming language will grow adoption over time. The <a href="https://zio.dev">ZIO community</a> continues to grow, but upon close inspection, the majority of this growth is occurring within the <em>existing</em> Scala market, and it is therefore not a proxy for the commercial success of Scala.</p>
<p>In light of this uncertain future, I believe that the time for action has come. As passionate commercial Scala developers, evangelists, and OSS contributors, we have seen how quickly the language helps us build reliable systems at scale. We know that Scala has the potential to take cloud-native programming to a new level of robustness, performance, and productivity.</p>
<p>In the ashes of Scala 2 and the Lightbend OSS ecosystem, we need a rebirth for Scala. A <strong>Scala resurrection</strong>, so to speak, which can help the language and ecosystem soar to new heights.</p>
<p>In this post, I will discuss the issues standing in the way of Scala’s growth in industry, and share my thoughts about how we can all help.</p>
<h2 id="language">Language</h2>
<p>With the introduction of Scala 3, the language has new life.</p>
<p>Scala 3 simplifies the Scala language in many important ways, such as splitting different use cases for implicits into special-purposed language features. The compiler itself has a more modern and maintainable architecture than its predecessor, which allows improvements and bug fixes to be made more quickly and more reliably.</p>
<p>Scala 3 is powerful, and while by no means perfect, it packs significant improvements in the areas of type-safety, metaprogramming, and abstraction, compared to all mainstream programming languages. Moreover, the language learns from mistakes made in Scala 2, and responds to some (but not all) feedback from commercial Scala developers.</p>
<p>Despite the promises, there remain challenges with Scala 3:</p>
<ol>
<li><strong>Academic bias</strong>. Scala has a history of accreting academically novel features that do not help or which actively hurt commercial adoption, and I believe this process is ongoing with Scala 3.</li>
<li><strong>Language instability</strong>. The Scala 3 language is essentially a different language than Scala 2, and the changes are of such size and scope that upgrading large code bases will take years, if it happens at all.</li>
<li><strong>Unproven compiler</strong>. The Scala 3 compiler is still beset by many bugs that have proven troublesome to work around, and which hint at the growing complexity of the compiler code base.</li>
</ol>
<p>Continued pain and attrition due to the Scala 3 upheaval is inevitable, but not fatal. With language and ecosystem stability, combined with tangible library benefits from upgrading to Scala 3, I believe we will eventually convince most serious commercial users.</p>
<p>However, I know that millions of euros are currently being invested into Caprese, which is a capability-based effect system built on capture tracking that could be a new language unto itself. If Caprese is transformed into a hypothetical Scala 4, it would upend not just the language, but the entire OSS ecosystem and all tooling, making the syntactical changes between Scala 2 and 3 seem tame by comparison.</p>
<p>The looming threat of a future radically different Scala 4 is the single greatest existential threat to Scala.</p>
<h2 id="tooling">Tooling</h2>
<p>In the old days, a text editor and <code class="language-plaintext highlighter-rouge">make</code> were considered sufficient “tooling”, but as programming languages have matured, so has their tooling, and expectations from developers are vastly higher today than at any point in history.</p>
<p>Every successful commercial programming language must include four key tools:</p>
<ol>
<li>A package manager and online registry.</li>
<li>A build tool.</li>
<li>An integrated development environment (IDE).</li>
<li>An interactive mode (REPL).</li>
</ol>
<p>A programming language might be able to acquire mainstream status with one deficiency. But achieving mainstream status with multiple deficiencies is extremely unlikely, given how far developer tooling has come.</p>
<p>Let’s look at how Scala stacks up in each area.</p>
<h3 id="package-manager">Package Manager</h3>
<p>Despite the existence of <a href="https://github.com/coursier/coursier">Coursier</a>, Scala essentially has no native package manager or online registry. Instead, owing to its roots on the JVM, Scala relies heavily on infrastructure common to the platform.</p>
<p>In the JVM, <em>repositories</em> hold build artifacts and dependencies. The structure of these repositories and the protocol by which they are accessed ultimately derives from <em>Apache Maven</em>, a project management tool built around objects like artifacts, versions, groups, and projects.</p>
<p>With Maven, there does not exist a <em>single</em> online repository of all public packages. Rather, there are <em>many</em> repositories, such as Maven Central, Sonatype Releases, JCenter, JBoss, and so forth. In addition, anyone can create and maintain their own repository, although build tools will not be able to fetch projects from custom repositories without additional configuration.</p>
<p>Relying on existing JVM infrastructure offers some advantages:</p>
<ul>
<li>The tooling around Maven repositories is incredibly rich, including commercial offerings for hosting private repositories, usage analytics, multiple client libraries in every JVM language for accessing repositories, support for Maven in every build tool, and so forth.</li>
<li>Java is a common port-of-entry for developers learning Scala.</li>
</ul>
<p>On the other hand, piggybacking on Maven comes with some serious drawbacks:</p>
<ul>
<li>Maven was not designed for Scala versioning and multi-platform support.</li>
<li>Publishing new projects on Maven is unreasonably complex and involved.</li>
<li>Maven is built on technologies of yesteryear, such as XML.</li>
<li>There is no one-stop-shop (<em>global online registry</em>) for all open source dependencies.</li>
</ul>
<p>Modern package managers such as Cargo or NPM are tailor-made for their respective languages. Publishing and consuming dependencies is simple, in many cases a <em>one-liner</em> that’s baked into standard tooling. JSON and REST APIs are supported natively. And there is a definitive online registry of open source projects that is pre-wired into all developer tooling.</p>
<p>Maven may have been an advantage in earlier, simpler times, but in this day and age, it’s a significant liability—a dinosaur from another era, which pushes people away from Scala.</p>
<p>Overall, package management in Scala gets a D, barely passing and that, based solely on Maven’s maturity and its familiarity to Java developers.</p>
<p>Grade: D</p>
<h3 id="build-tool">Build Tool</h3>
<p>Except for an IDE or text editor, there is no tool a developer spends more time with than a build tool: the tool responsible for running compilers, downloading dependencies, and assembling artifacts that can actually be executed or deployed.</p>
<p>A good build tool can keep developers happy and productive, all the way up from simple “Hello World” applications to complex, large-scale applications that are split across dozens and dozens of projects, with hundreds of dependencies and very custom build steps.</p>
<p>The de facto build tool for Scala is <em>Simple Build Tool</em>, better known by the acronym <em>SBT</em>. For well more than a decade, SBT has gotten the job done, providing a way to build small and large Scala projects, both simple and complex.</p>
<p>To its credit, SBT did not shy away from complex builds, unlike many other attempts at creating build tools for Scala. Furthermore, its wide adoption is a testament not merely to its first-mover advantage, but to its features, documentation, build model, and extensive ecosystem of plugins. I and many other Scala developers owe a huge debt of gratitude and appreciation for the countless developer years poured into this tool, both by individual contributors who generously donated their free time, as well as by companies.</p>
<p>Unfortunately, <a href="https://www.lihaoyi.com/post/SowhatswrongwithSBT.html">SBT falls far short</a> of the standards for modern build tools. Indeed, SBT falls so far short that numerous frustrated developers over the past decade have attempted to write better build tools for Scala (Mill, CBT, Seed, Fury, Bleep, etc.).</p>
<p>At a high-level, SBT suffers from the following major drawbacks (among others):</p>
<ul>
<li><strong>Extremely complex</strong>. SBT has a high ramp up curve. When faced with a new SBT project, a developer starts with a terrifying empty Scala file, and must know what sequence of characters to type in order to create a “Hello World” project (there are workarounds, but the fact that these workarounds exists is proof of the massive developer-experience problem).</li>
<li><strong>Neither code nor data</strong>. Data-based build tools, such as Ant, are purely declarative, and can be analyzed, optimized, turned into documentation, and easily generated and consumed by other tools; code-based build tools allow developers to write ordinary code to build their projects. SBT is neither data-based nor Scala-based, instead choosing a weird Scala DSL that has none of the advantages of either approach.</li>
<li><strong>Legacy monolith</strong>. SBT was written over more than a decade, resulting in layers of aging code, an overly complicated architecture, and completely different, incompatible styles of Scala, making it more difficult to attract contributors.</li>
<li><strong>Poor Scala integration</strong>. Despite being written in Scala and targeting Scala builds, SBT has no deep understanding of Scala versioning or cross-platforming, or the Scala compiler itself, and leaves many of the hard problems of building complex Scala applications to developers.</li>
<li><strong>Inscrutable failures</strong>. Diagnosing problems with SBT can be extremely frustrating. To this day, there are times when SBT will fail to build a project, leading to the loss of countless developer hours. I myself once walked away from fixing a build problem after spending 40 hours without progress.</li>
</ul>
<p>In my opinion, the main reason developers choose SBT to build Scala projects is because developers choose SBT to build Scala projects. For all these drawbacks, Scala’s build tool gets an F.</p>
<p>Grade: F</p>
<h3 id="ide">IDE</h3>
<p>An Integrated Development Environment (IDE), once a luxury, is now standard for all modern mainstream programming languages. With the most popular programming languages, developers have a choice of multiple IDEs—some commercial, some open source; some desktop-based, some cloud-based.</p>
<p>Thanks to type- or receiver-directed auto-complete, inline code documentation, inline errors and warnings, intelligent code refactoring, go-to-definition, and many more features, IDEs greatly increase the productivity and joy of developers, making many developers refuse to go back to editing code files with primitive text editors (whose only tool is “copy and paste”).</p>
<p>In recent history, Scala developers have had a choice of three IDE-like experiences:</p>
<ul>
<li><em>IntelliJ IDEA</em>, with the Scala language plug-in</li>
<li><em>Visual Studio Code</em>, with <em>Scala Metals</em></li>
<li><em>Emacs</em> with <em>Ensime</em></li>
</ul>
<p>Only one of these would claim to be an IDE (IntelliJ), and only the first two have roughly the right feature set for an IDE.</p>
<p>Before evaluating these solutions, a few words of praise are in order.</p>
<p>JetBrains deserves kudos for building the finest Java IDE I have ever used (I happily paid for licenses in my Java days). I’m also quite grateful that JetBrains has invested in so many language plug-ins. Similarly, I’m grateful to the Scala Metals team for bringing me the first solution that I felt like <em>could become</em> the IDE that I wanted for Scala.</p>
<p>Many talented and smart developers poured much laudable effort into both of these solutions.</p>
<p>Yet both IntelliJ IDEA for Scala and Scala Metals suffer drawbacks so serious, they are more prototypes than products, and the distance between them and something that delivers the experience of IntelliJ IDEA for Java is <em>growing</em>, not <em>shrinking</em>.</p>
<p>For quite plausible-sounding reasons, IntelliJ IDEA made a gigantic mistake when they decided to implement their own syntax, type, and kind-checking for the Scala plug-in. This same decision for Java resulted in an amazing IDE that could function incrementally and in the presence of type errors (it’s quite useful to refactor code even when it doesn’t fully compile).</p>
<p>However, Java has a very simple type system that is static and well-defined. Scala’s type system is extremely complex and defined only by the implementation, which itself has numerous quirks, edge cases, and of course bugs. Moreover, the semantics, quirks, edge cases, and bugs all vary somewhat between minor versions of the Scala compiler, and are completely different between major versions. Attempting to emulate Scala’s type system means that IDEA and Scalac often do not agree on whether code compiles, and if it does compile, what the error message is.</p>
<p>Rather than correct this mistake by using the Scala compiler itself for errors and warnings, IntelliJ IDEA has doubled-down on the mistake, time and time again. Perhaps their next-generation editor <em>Fleet</em> is an acknowledgement both of the fact that JVM desktop applications are truly dead, and duplicating independent syntax, type, and kind-checkers for every language is not viable.</p>
<p>The architectural issues with the IntelliJ IDEA Scala manifest themselves in ways that frustrate Scala developers: code that compiles with <code class="language-plaintext highlighter-rouge">scalac</code> does not compile with IntelliJ IDEA. Code that does not compile with <code class="language-plaintext highlighter-rouge">scalac</code> compiles just fine with IntelliJ IDEA. One never knows if one’s code is truly compiling without running the build tool, which defeats the primary purpose of an IDE.</p>
<p>Beyond these frustrating issues with simple compilation, the Scala plug-in for IntelliJ IDEA simply is not up to the same standards as their Java IDE. Refactoring support is barely present, and doesn’t always work. With so many resources spent merely trying to keep the plug-in’s model of Scala consistent with all the editions of the two Scala compilers, it is hardly surprising the team can’t implement other IDE features.</p>
<p>As a result, while many commercial Scala developers still use IntelliJ IDEA (because, for all its drawbacks, it is significantly more stable and robust than Scala Metals), it is not an acceptable development experience, and it is no where in the same league as modern IDEs for languages such as Java, Kotlin, C#, Go, TypeScript, or even Python.</p>
<p>My own frustrations with IntelliJ IDEA led me to try Scala Metals, and I was blown away by the early experience of using the tool: with just a few clicks, it would compile my project and start reporting errors in VS Code, which were powered directly by the real Scala compiler.</p>
<p>Yet having used Scala Metals for a few years now, I know well that even though Metals demos well in controlled environments, the experience of using Metals in production is extremely frustrating. Quite simply, Metals is an <a href="https://github.com/scalameta/metals/issues/4837">extremely buggy piece of software</a>, which frequently gets into a state where it completely stops working.</p>
<p>In fact, so common is the experience of Metals breaking that some developers have made scripts to blow away all of its directories, kill its JVM processes, and restart Metals in hopes the incantation will work again for some (probably short!) period of time.</p>
<p>Beyond the painful experience of using Metals due to its longstanding instability, Metals has chosen an architecture which is fundamentally flawed. Although Metals does not have its own syntax, type, and kind-checker (like IntelliJ IDEA for Scala), Metals invented its own “Build Server Protocol” and a separate build engine called Bloop for understandable but ultimately technically unjustifiable reasons.</p>
<p>So not only does Scala Metals deliver a very poor experience, but it’s built on a suboptimal architecture—a foundation that is far too large, and which will continue to accrete technical debt at an unsustainable rate.</p>
<p>In short, there are no IDEs for Scala right now with a modern experience suitable for commercial software development, nor any obvious path to a commercial-grade IDE. Consequently, Scala’s grade for IDE support is a definite F.</p>
<p>Grade: F</p>
<h3 id="repl">REPL</h3>
<p>The REPL, which stands for Read-Eval-Print-Loop, allows developers to explore a language’s syntax, semantics, and libraries, and to solve simple programming problems in an interactive way that maximizes feedback and encourages iteration and experimentation. While often targeted at new developers, REPLs are often used by seniors.</p>
<p>To its credit, Scala had a REPL before it was cool for statically-typed languages to include one. The REPL grew increasingly outdated over the years, failing to keep up with modern REPLs (especially those of dynamically-typed programming languages).</p>
<p>Then <a href="https://ammonite.io/">Ammonite</a> came along, and added numerous features, such as syntax highlighting, multi-line editing, dependency loading, and much more, giving Scala a much more impressive REPL story, albeit one laden with edge cases in the wake of Scala 3.</p>
<p>Overall, because Scala does have a modern REPL in Ammonite, but because Ammonite does not entirely support Scala 3 and the project is not integrated into the Scala compiler, Scala’s REPL support gets a grade of B.</p>
<p>Grade: B</p>
<h3 id="tooling-summary">Tooling Summary</h3>
<p>Scala’s average rating is a D, well below what I would consider the <em>bare minimum</em> for a mainstream programming language. Moreover, this grade represents a lowering from the peak of the Scala 2 lifecycle, where tooling and in particular IDE support reached an all-time high.</p>
<p>It is worth pointing out that while these are my own ratings, they are <a href="https://scala-lang.org/blog/2022/12/14/scala-developer-survey-results-2022.html">consistent with the views of the broader Scala community</a>, at least as collected by the non-random (and therefore certainly biased) 2022 Scala survey.</p>
<p>The state of tooling represents the most immediate and greatest challenge to working with the Scala programming language today.</p>
<h2 id="ecosystems">Ecosystems</h2>
<p>Scala has multiple open source ecosystems, which target specific pain points in building backend applications. In the sections that follow, I overview the strengths and weaknesses of each, with an eye toward how these ecosystems affect Scala adoption.</p>
<h3 id="akka">Akka</h3>
<p>The Akka ecosystem originally intended to bring Erlang-style actors to the JVM. Through excellent marketing, and partially by riding the early Scala hype train, Akka managed to find itself over-hyped and over-deployed, with many companies later needing to rip out the technology (quite simply, actors are a terrible solution for local concurrency).</p>
<p>As Akka matured, however, and valid use cases became clear (distributed, persistent actors and an async foundation for local streaming), the ecosystem found its niche and its stride, and was successfully adopted by many large companies in mission-critical use cases.</p>
<p>These days, however, Lightbend, the company behind Akka, decided to monetize the framework directly, by changing from an open source license to the <em>Business Source License</em>, which had proven monetizable for databases and other developer infrastructure.</p>
<p>Although there is an open source fork of Akka called Pekko, it is not well-maintained and stands on questionable legal foundations. Thus, every company I have talked to about Akka has told me they are ripping it all out. Apache Flink, which depended on Akka, also plans to rip it out, with no plans to depend on Pekko.</p>
<p>Akka is <em>still</em> a powerful framework for building distributed and streaming applications. However, I do not expect Akka to help grow the Scala ecosystem, and in fact, I expect the change in licensing to push some companies away from Scala.</p>
<p>Thus, with a Scala-centric lens, Akka gets a grade F, solely for its negative effects on Scala adoption.</p>
<p>Grade: F</p>
<h3 id="zio">ZIO</h3>
<p>The ZIO ecosystem has brought a fresh and innovative fusion of practicality, ergonomics, and functional programming to the domain of building scalable, bulletproof cloud applications.</p>
<p>Ported to multiple programming languages, and even inspiring a few companies to “Go Scala”, ZIO has become widely adopted on its own merits, while providing a partial but increasingly viable alternative to Akka, Spring, and other modern application development frameworks.</p>
<p>The ZIO community actively creates many new open source contributors and speakers each year, mentors existing open source contributors, and creates and maintains integrations to other Scala ecosystems, without any politics and with old school professionalism.</p>
<p>Despite all the good that the ZIO ecosystem is doing for Scala, however, the ZIO ecosystem has multiple weaknesses:</p>
<ul>
<li><strong>Incomplete stack</strong>. As of the date of this writing, the ZIO ecosystem does not have a complete stack for backend development, its major weaknesses lying in HTTP (there is no 1.0 version of <em>ZIO HTTP</em>), persistence (only <em>ZIO Quill</em> is production-grade, and not everyone likes LINQ-style database access), and gRPC (there is a great library but performance needs to be optimized).</li>
<li><strong>Early-grade projects</strong>. The ZIO ecosystem contains a number of projects that are still early-stage (<em>ZIO JDBC</em>, for example), and which require continued investment in order to become production-grade.</li>
<li><strong>Limited documentation</strong>. While the core ZIO project is quite well documented at this point, many critical ecosystem projects are not documented at the same level. In some cases, there is only a page or two of documentation.</li>
</ul>
<p>Given these drawbacks, together with the positive aspects, the ZIO ecosystem receives a grade of B.</p>
<p>Grade: B</p>
<h3 id="typelevel">TypeLevel</h3>
<p>The TypeLevel community, which started life as a political fork of the much older Scalaz (the first port of Haskell libraries to Scala), attempts to bring category-theoretic functional programming to commercial software development in Scala.</p>
<p>Being the oldest still viable example of “functional programming in Scala”, the TypeLevel ecosystem has slowly developed a full-stack solution for the backend, including solutions for persistence, HTTP, logging, and caching. With much better documentation than Scalaz ever had, TypeLevel has also made Haskell-style functional programming accessible to more Scala developers than ever, showing commercial software developers the practical benefits of referential transparency and algebraic abstractions.</p>
<p>While having a positive impact on adoption of functional Scala within industry, the TypeLevel ecosystem has weaknesses of its own, including:</p>
<ul>
<li><strong>Developer-experience</strong>. Numerous developers have shared feedback on the challenges of using some TypeLevel libraries such as http4s and Cats Effect, particularly for those new to Scala and functional programming. Some new TypeLevel contributors have pushed for improvements (successfully, in the case of Cats Effect documentation, for example), but generally such feedback is not welcomed or labeled FUD, and leads to responses like “Get Educated”.</li>
<li><strong>Community challenges</strong>. TypeLevel has had its share of community challenges, partly due to its controversial founder <a href="/articles/travis-brown-abuser">Travis Brown</a>, a known abuser, cyber-stalker, and harasser who was forced out of the Scala community, but also due to the actions of a tiny minority of TypeLevel members, the personal friends of Travis Brown—actions like supporting Travis Brown, deleting reports of CoC violations, engaging in elitist putdowns of OSS work deemed “insufficiently pure”, smearing the character of other OSS contributors, and all-around unprofessional behavior toward ZIO, ZIO integrations, and ZIO contributors.</li>
</ul>
<p>In light of both the positive impact but also TypeLevel’s challenges around usability and community, the ecosystem receives a grade of B.</p>
<p>Grade: B</p>
<h3 id="other">Other</h3>
<p>Though Scala is home to many open source libraries, most of these are not part of any ecosystem. The two partial exceptions are SoftwareMill’s range of libraries, including Tapir and sttp, and Li Haoyi’s Python-esque libraries that focus on getting things done with a minimum of ceremony.</p>
<p>There is no doubt many of these libraries are high quality, especially in the aforementioned two ecosystems. Where they are present, weaknesses of libraries across the greater Scala ecosystem include:</p>
<ul>
<li><strong>Duplicated effort</strong>. For its relatively small size, Scala has a disproportionately high number of libraries all trying to solve any given problem. Given that Scala is not very opinionated, but that OSS contributors very often <strong>are</strong> opinionated, perhaps this is not surprising.</li>
<li><strong>Abandoned libraries</strong>. Due to their origin as one-person passion projects, many older Scala libraries are abandoned or in a state of disrepair, without being officially deprecated or transitioned to more motivated maintainers.</li>
<li><strong>Relatively unambitious</strong>. There are few concerted efforts to build large-scale open source systems that could rival the size and scope of projects like Flink, Spring, or even Akka (with some <a href="https://github.com/JohnSnowLabs">exceptions</a>).</li>
</ul>
<p>Overall, Scala gets a B in this category, an above average score. For its size, Scala has a wonderful third-party OSS ecosystem.</p>
<p>Grade: B</p>
<h3 id="ecosystem-summary">Ecosystem Summary</h3>
<p>Scala’s modern OSS ecosystem suffered a blow when Lightbend took the Akka ecosystem proprietary. Akka provided a comprehensive toolkit for building resilient, distributed, cloud-native applications, and showed the world that Scala offerings could compete with the very best of what other languages offered.</p>
<p>Despite these tumultuous changes, the Scala open source ecosystem remains vibrant and powerful—definitely playing in a league well beyond what the size of the Scala community would suggest.</p>
<p>However, despite being above average, there is ample room for improvement, in ZIO, TypeLevel, and other ecosystems. With improvement in a few core areas, Scala could unquestionably have one of the finest open source ecosystems of any programming language.</p>
<h2 id="platform">Platform</h2>
<p>We live in a multi-platform age. Our applications must run locally on different desktop operating systems, remotely on different server operating systems, on different virtual machines, and sometimes even in browsers or on the edge.</p>
<p>In this section, I will overview Scala’s support for targeting multiple platforms.</p>
<h3 id="jvm">JVM</h3>
<p>The JVM was the first platform that Scala supported, and to this day, it remains the only officially supported platform for Scala, and the only one that is baked into the compiler and broadly assumed by ecosystem tooling and libraries.</p>
<p>Despite the JVM being the first-class platform for Scala, there are still some major deficiencies in Scala’s support for the JVM:</p>
<ul>
<li><strong>One-way interop</strong>. It is virtually impossible for Java or Kotlin code to call into Scala code. The fact that other languages cannot consume Scala libraries diminishes the interop argument for Scala-on-JVM.</li>
<li><strong>Lackluster optimizer</strong>. Scala is a higher-level language than Java. Rather than trying to minimize the cost of abstraction, the Scala compiler generates dumb code that executes more slowly in the presence of more abstraction.</li>
<li><strong>Lagging support</strong>. Scala’s support for newer versions of Java, as well as new JDK or JVM features, lags behind. When the JVM gets value objects, for example, it is quite likely Scala support will severely lag; or as another example, the Scala compiler itself has been known to crash on modern JVMs.</li>
</ul>
<p>Because Scala does support the JVM natively and directly, but also because its support is not best-in-class, Scala’s support for the JVM platform gets a grade of C (average).</p>
<p>Grade: C</p>
<h3 id="js">JS</h3>
<p>Supporting the Javascript platform for Scala has always puzzled me somewhat. Currently, Javascript is used primarily for web front-ends, and most Scala developers have neither interest nor skills in frontend web development.</p>
<p>Most web frontend developers use Javascript or TypeScript. They do not want to learn a backend programming language to write frontend code.</p>
<p>That said, it’s quite nice to use the same language for both frontend and backend (if you are familiar with both), and doing so can often reduce code duplication. In addition, providing Scala support for the Javascript platform allows backend engineers to create and maintain applications that they would ordinarily have no interest in creating or maintaining.</p>
<p>The quality of the Scala.js platform is high. Sporting a small subset of the JDK, one can compile real world code on Scala.js, and get high-quality Javascript output. Method overloading and other Scala features are mostly emulated in the way you would want.</p>
<p>That said, however, Scala.js has a number of weaknesses:</p>
<ul>
<li><strong>Standalone target</strong>. Scala.js is not integrated into the compiler, but must be added as a compiler plug-in, with library dependencies.</li>
<li><strong>Complex build</strong>. Setting up a project for cross-building is difficult (with quite painful developer experience), complicates the build, and significantly slows down building and testing.</li>
<li><strong>Incomplete JDK support</strong>. The JDK is so vast that inevitably some class is needed which is not emulated in Scala.js.</li>
<li><strong>Poor tooling</strong>. The IDEs that we have do a poor job supporting Scala.js; one can argue this is a problem with the tooling, but it affets Scala.js users.</li>
</ul>
<p>Considering both the strengths and weaknesses of Scala’s Javascript platform support, Scala.js gets a grade of B.</p>
<p>Grade: B</p>
<h3 id="llvm">LLVM</h3>
<p>Scala’s LLVM support is known as <em>Scala Native</em>, because the end result of using LLVM is <em>native</em> platform-specific executables that run without bytecode-based interpretation or just-in-time compilation.</p>
<p>Scala Native is an attempt to wean Scala from its long-standing addiction to the JVM, providing a viable alternative platform for backend applications. JVM applications suffer from high-memory usage, GC pauses, and lengthy application startup and warmup times, all of which prove problematic with many classes of applications (command-line applications, cloud-native applications, etc.).</p>
<p>Scala Native is an enormous undertaking, because LLVM is much lower-level than the JVM or Javascript platforms. As with Scala.js, Scala Native must provide an emulation of some JDK constructs for virtually any program at all to run. But more than that, Scala Native must provide memory management, multithreading, and many other functions that we take for granted on the JVM platform.</p>
<p>A number of highly talented engineers have poured their own personal time and effort into Scala Native. In addition, the project has seen some modest investment by EPFL and the Scala Center.</p>
<p>Despite all of this investment, however, Scala Native is <em>no where near</em> to a suitable replacement for Scala JVM. The vast majority of Scala applications are not even close to <em>compiling</em>, let alone executing reliably and performantly, on Scala Native. And as with Scala.js, the tooling that exists or at least partially works for the JVM is missing or buggy for Native, leading to a poor development experience.</p>
<p>As much as I see the promise of viable alternatives to the JVM, Scala’s LLVM support receives a grade of D, simply because it is not there yet, and requires significant further investment to become viable for most JVM-based Scala applications.</p>
<p>Grade: D</p>
<h3 id="wasm">WASM</h3>
<p>WASM has emerged as a possible successor to the JVM specification, one built for a cloud-native era, in which sandboxing is the norm, and low-latency, low-memory consumption cloud apps rule the skies.</p>
<p>In theory, there is a way to translate from LLVM IR to WASM through Emscripten. In practice, this is so far away from anything usable for production that Scala gets an F for WASM support.</p>
<p>Grade: F</p>
<h3 id="platform-summary">Platform Summary</h3>
<p>Scala’s platform strengths reside squarely with the JVM, which is unsurprising given the origin of the language. What is increasingly surprising, however, is that Scala remains an essentially JVM-only language.</p>
<p>Scala.js is an excellent attempt to bring Scala to the front-end via Javascript, albeit with a limited market and challenges around JDK support. Scala Native (LLVM), on the other hand, while being a remarkable achievement, is such a vast undertaking that it’s too early, too limited, and too immature to benefit the typical Scala application. The LLVM pathway to WASM (if you want to call it that!) is merely a hello-world toy.</p>
<p>If you believe the JVM is, without question, the future platform for cloud- and edge-native applications, then Scala’s existing platform support may be everything you want and need. For those hoping for native applications or a more modern take on a cross-platform virtual machine, Scala’s current cross-platform support leaves a lot to be desired.</p>
<h2 id="scala-resurrection">Scala Resurrection</h2>
<p>The difference between a hater and a lover is that a hater shouts (destructive) criticism to tear down, while a lover offers (constructive) criticism to build up.</p>
<p>To go as far as I believe the language can go, Scala needs building up. Not from just one individual, such as myself, or even Martin Odersky. Rather, <em>Scala needs all our help</em>, as we work together to make the language as commercially successful as it deserves to be.</p>
<p>We need a coordinated campaign to rectify known weaknesses, and shore up emerging strengths. Such a campaign will not be easy. It will require discipline, collaboration, and sacrifice, acting with a sense of common purpose for the greater good of all.</p>
<p>In the sections that follow, I’m going to present what I think needs to happen, as well as a few ideas on how it could happen. If you disagree or have your own suggestions, then please post below!</p>
<h3 id="industry-language">Industry Language</h3>
<p>In my opinion, mainstream adoption of Scala requires a commitment to language stability. There can not be more “Scala 3” editions, which re-imagine the semantics and syntax of the core language, break virtually every library ever developed, reset tooling to ground zero, and require a combination of automated and manual code rewrites.</p>
<p>It is absolutely true that Scala 3 is not a perfect language. It is also true that Java is not a perfect language. Yet Java remains backward compatible with the very first Java program ever written, which allows the ecosystem to continuously improve, and which gives companies confidence in investing in the Java programming language.</p>
<p>Scala has been a useful vehicle for exploring academically interesting ideas, such as the dot calculus which forms the basis of Scala 3. But in order to enter mainstream, academic experiments must live outside Scala. They must live in new languages, or new dialects of Scala that are well insulated from the commercial edition of Scala.</p>
<p>Scala now needs a firm commitment to maintain backward compatibility indefinitely. We need to be able to run the same Scala 3 programs now that we can run 10 years from now, without having to completely rewrite them (with or without partial automation).</p>
<p>This commitment might mean that new academic explorations might not make it into Scala, but it will also mean Scala’s ecosystem (particularly its tooling, but also libraries) will bloom, and companies will have the confidence they need to invest seriously into the language.</p>
<p>Scala also needs a commitment to fixing bugs and improving performance in the core compiler. Scala developers need a compiler that works reliably and performantly, and while that’s true in many cases, there is much opportunity for ongoing improvement.</p>
<p>Finally, Scala needs work on an optimizer that can minimize the cost of abstraction. There is little value in having a language as expressive as Scala if actually taking advantage of its full powers slows applications down. And there is no reason why it should have to, as most forms of abstraction can be eliminated via a sufficiently advanced optimizer.</p>
<h3 id="world-class-tooling">World-Class Tooling</h3>
<p>Scala desperately needs an IDE that works reliably across complex, real world projects. An IDE is table-stakes for any mainstream programming language, and while IntelliJ IDEA and Scala Metals have both tried to give Scala developers the IDE they need, neither has yet attained that goal, and architectural choices likely prevent these projects from ever attaining this goal.</p>
<p>We need an ultra lean, lightweight architecture for a squeaky clean IDE that can be maintained by a small team with an exceptionally low defect count. Far more important than breadth of features is core stability and developer experience. By leveraging existing open source, by learning from the architectural mistakes of IDEA and Metals, we can engineer a new and extensible solution that delivers the joyful and productive experience that modern developers expect from their IDEs.</p>
<p>Scala also needs a modern build tool, and can chart one of three paths to get there:</p>
<ol>
<li><strong>Invest in multi-build tool</strong>. Multi-build tools like Pants, Bazel, etc., can be used for building any kind of project. Although support for Scala is weak in such build tools, it could be improved with further investment from the Scala community.</li>
<li><strong>Invest in the most promising SBT alternative</strong>. None of the SBT replacements are currently viable for all use cases being served by SBT. However, if the authors are interested and open to collaboration, it’s possible some of these build tools could be improved to handle the full range of use cases handled by SBT, but without SBT’s many drawbacks.</li>
<li><strong>Develop a new build tool</strong>. Although this seems like a herculean task, modern Scala OSS includes libraries like ZIO Streams and FS2, which, together with smaller, special-purpose libraries like Coursier, could dramatically reduce development time.</li>
</ol>
<p>Though we can wait on this one for a while, Scala still needs its own package manager, distinct from Maven, which is built for Scala libraries and applications, which knows about Scala’s versioning and cross-platform support, and which provides the joyful experience required by new developers, and the extensible experience required by seasoned professionals.</p>
<p>Scala also could benefit from a first-class REPL experience like Ammonite, polished, extended with full support for Scala 3, and incorporated directly into the official Scala compiler, as a replacement for its existing REPL.</p>
<p>In discussing tooling, I must mention the work that Virtus Lab has done in creating <a href="https://scala-cli.virtuslab.org/">Scala CLI</a>, a simple way to compile, run, test, and package single-module Scala projects that was recently accepted as the new <code class="language-plaintext highlighter-rouge">scala</code> command.</p>
<p>While not really a build tool right now (for most real world projects), Scala CLI could potentially become a foundation for a build tool, with package management and integrated REPL, as well as IDE support.</p>
<p>Solving the tooling challenges will require investment into promising open source projects, investigating ways to integrate the best parts from multiple OSS projects (for example, Scala CLI, Ammonite, Fury, Ensime), and the creation of new open source projects. Moreover, it will require a commitment to ongoing, broad-based investment from the community at large, as well as partners from academia and industry.</p>
<h3 id="healthy-ecosystem">Healthy Ecosystem</h3>
<p>The Scala open source ecosystem is strong, but needs to become stronger. Obviously, the respective ecosystems can and should embrace their respective strengths, while addressing their respective weaknesses.</p>
<p>However, I believe Scala can go much further than just ecosystem-siloed improvement.</p>
<p>With a finite resources, we need to make every hour of open source labor count. One of the most rational and effective ways to do this is to reduce duplication across Scala’s open source ecosystems.</p>
<p>Does Scala really need 20 JSON libraries, 5 schema libraries, 9 CLI libraries, 4 effect system libraries, and so on? Probably not. We need to consolidate, and rally around strong projects; merge our strengths, cancel our weaknesses.</p>
<p>There’s probably a way to do that, in some cases, that makes everyone happy.</p>
<p>Let’s say we wanted to consolidate JSON libraries. Each ecosystem having a JSON library could appoint a person to collaborate, who would be an admin in a joint project. The best aspects of each library could be combined into a single library that meets the needs of all users.</p>
<p>In support of this goal, I will personally commit to helping organize a <em>Scala Open Source Summit</em>, which encourages all ecosystems to come together to discuss and hack on Scala OSS. This should help foster dialogue between different ecosystems to see if common ground can be found.</p>
<p>If the most brilliant minds in Scala open source work together, then surely, the whole ecosystem can rise to new heights!</p>
<h3 id="natively-cross-platform">Natively Cross-Platform</h3>
<p>The JVM is attempting to innovate, and to its credit, is doing so faster now than at any time in its history. However, the world has moved far past the JVM, with cloud-driven requirements including auto-scaling, fast startup times, and low memory usages, running on heavily virtualized and sandboxed infrastructure, for applications demanding increased efficiency for greater cost savings.</p>
<p>The JVM doesn’t have a solution for these problems, and while Oracle can solve this problem, it’s not clear they will. There is obvious tension between Oracle’s desire to improve the JVM, and Oracle’s desire to make money on the JVM, and some of the pain points in the JVM are already being monetized (see <em>GraalVM Enterprise</em>).</p>
<p>If we were to redo the JVM specification from the ground-up, it might look a lot more like the WASM specification. WASM points to a future where a new operating system or perhaps a single application on an existing operating system runs hundreds or thousands of WASM applications in a completely secure, sandboxed fashion, at low cost, with auto-scaling and low memory usage.</p>
<p>In any case, it’s clear to me that sticking with the JVM is an increasingly risky strategy. It ties the fate of Scala to the fate of the JVM. But Scala doesn’t need the JVM to be a powerful, compelling solution to building modern applications.</p>
<p>Maybe in the early days, Scala needed the JVM. But no longer.</p>
<p>Scala needs to support alternative platforms, and it needs to do so directly, integrated into the compiler. The JVM platform should not be specialized. Rather, each backend should take an intermediate representation (perhaps TASTY, or perhaps something lower-level) and generate platform-specific code, sharing optimization passes where it makes sense to do so.</p>
<p>A huge liability in supporting multiple platforms is the Scala ecosystem’s heavy reliance on the JDK. Both Scala.js and Scala Native have to re-implement the JDK, which is a monstrous task to do exhaustively and correctly. Moreover, it’s not even really necessary, since an increasing percentage of Scala applications are not written to the JDK, per se, but to Scala’s own open source ecosystem.</p>
<p>In order to enable low-cost, bulletproof cross-platform support, Scala <em>needs</em> a low-level, cross-platform library that provides the bare-bones essentials for platform support: including threads, files, sockets, processes, and rudimentary time functionality (one should look to various preliminary WASM specs for inspiration).</p>
<p>Scala’s vast OSS ecosystem would then adopt this cross-platform library to power their features.</p>
<p>In this fashion, Scala applications depending on Scala’s own open source ecosystems would instantly become compatible with all of Scala’s supported platforms, to the maximum extent possible. In addition, there would cease being such a thing as “cross-platform libraries” (which are hell to maintain), because all libraries would always be fully cross-platform.</p>
<p>Such a cross-platform library could be built by volunteers, and architected and overseen by leaders from all the major OSS ecosystems in Scala (since ultimately, they would be its direct users, and critical to weaning Scala applications off the JDK).</p>
<p>As CEO of Ziverge and a core contributor to ZIO, I am happy to play a role in developing a low-level cross-platform library, but we need key stakeholders from other ecosystems and companies in the space to rally behind the idea.</p>
<h3 id="the-how">The How</h3>
<p>As the Scala programming language and broader ecosystem is not a commercial product, there is no single entity who can fund development and maintenance of critical infrastructure. Although in some cases, I’ve volunteered my own time and resources, or those of my company, ultimately, in order to address all of these challenges, we need much broader participation.</p>
<p>Fortunately, there are many organizations who can and do contribute to Scala. The involvement of some of these organizations will be crucial.</p>
<h4 id="lamp">LAMP</h4>
<p>LAMP is the <a href="https://www.epfl.ch/labs/lamp/">lab at EPFL</a> that is run by Martin Odersky, and which contributes directly to Scala 3 and the broader ecosystem, including open source and tooling.</p>
<p>LAMP receives its funding from grants (such as the Advanced Grant program from the Swiss National Fund), which are targeted at research and development in specific areas of programming methods.</p>
<p>Due to the way that grants to LAMP work, there is pressure on projects to produce research that is academically novel. Yet, academic novelty is frequently at odds with the needs of industry: commercial developers want languages, tooling, and open source to solve practical problems in straightforward ways, without regard to academic novelty, theory, or value to the research community.</p>
<p>Beyond this basic tension, many projects coming from LAMP are spearheaded by PhD students, who, when they obtain their doctorate, abandon their project and accept a job in industry, creating a steady stream of half-baked abandonware that never realizes its full promise in industry.</p>
<p>Despite these drawbacks, there is no doubt that LAMP has powered tremendous positive change for the Scala programming language, including the Dot calculus which brings a sounder foundation to Scala’s advanced type system. The students and assistants working at LAMP are quite brilliant and hard working, but these traits do not solve the tension between academic research and industry needs, nor address the finite tenure of students.</p>
<p>To improve upon its legacy, I believe LAMP could benefit from a new fusion of industry and academia:</p>
<ol>
<li><strong>Broad industry input into research projects</strong>. There are many possible improvements to the Scala programming language that are both academically interesting, as well as incredibly useful to industry (for example, GC-less allocation and de-allocation through whole program analysis; stack-based storage through local primitive record inlining; lightweight linear typing for leak-proof resources; etc.). If the industry has some input into research projects, it would help ensure they contribute to the commercial success of Scala.</li>
<li><strong>Industry partnership to prevent half-baked abandonware</strong>. Ideally, there would be no LAMP projects targeting the Scala language or ecosystem that do not have an industry partner, who commits to the maintenance of the project after completion. Grant money is non-recurring, and PhD’s are one-and-done, so Scala artifacts produced through LAMP need industry partnership to be maintained or polished into commercial grade technology.</li>
</ol>
<p>By directly involving industry in the development and maintenance of projects, the work that LAMP does will be more relevant than ever, have a much broader impact on commercial software development, and result in long-lasting benefits for the entire Scala community.</p>
<h4 id="scala-center">Scala Center</h4>
<p>The Scala Center is a not-for-profit organization backed by EPFL and corporate sponsors, dedicated to open source and education around the Scala programming language. The organization is closely connected to and shares some of the same leadership and team as LAMP.</p>
<p>As with LAMP, the Scala Center has made many positive contributions to the Scala programming language and ecosystem. Originally formed to fill the void in the Scala ecosystem after Lightbend (Typesafe) pivoted away from Scala, the Scala Center has contributed to Scala.js, Scala Metals, Scalafix, Scalafmt, and many other projects.</p>
<p>While numerous smart and talented engineers have worked at Scala Center on important projects, the organization itself has had major challenges, including:</p>
<ul>
<li><strong>Re-inventing wheels</strong>. Rather than embrace and extend existing open source projects, Scala Center has chosen to build new open source projects (or even new protocols such as BSP!). This can alienate existing OSS contributors (as it did with the author of Ensime), creates more duplicated work, and increases chances of failure due to fewer committed stakeholders.</li>
<li><strong>Abandoning projects</strong>. Scala Center has a long history of creating projects that show potential and then abandoning them.</li>
<li><strong>Breadth over depth</strong>. Scala Center has a demonstrated history of preferring “feature count” to depth and production-worthiness.</li>
</ul>
<p>While these problems are not particularly hard to solve, they do require effective organization and management, collaboration with industry partners and open source contributors, and a commitment to long-lived and boring (but essential!) open source maintenance.</p>
<p>Instead, at a time when Scala tooling has fallen behind because of Scala 3, the Scala Center chose to spend resources building an <a href="https://www.scala-lang.org/blog-detail/2022/04/05/inclusive-language-guide.html">inclusive language guide</a>. Personally, I don’t know any Scala developers who would prefer an inclusive language guide over a working IDE.</p>
<p>Worse still, at a time when the language, tooling, and cross-platform support require more financial investment than ever before, the Scala Center rejected a generous offer of corporate sponsorship from <a href="https://ziverge.com">Ziverge</a>.</p>
<p>As a CEO, I know how to fix these problems, but unfortunately, there is nothing any of us can do because there is no democratic or open mechanism for influencing the organization, management, or priorities of the Scala Center.</p>
<p>In the best case scenario, the Scala Center continues to operate as before, which means the Scala community gets <em>some</em> value from the organization, but cannot rely on the Scala Center to move the needle of adoption or completely solve any of the challenges Scala has.</p>
<h4 id="commercial-organizations">Commercial Organizations</h4>
<p>The main commercial organizations involved in contributing to the greater Scala ecosystem include:</p>
<ul>
<li><strong>JetBrains</strong>. JetBrains is investing quite a lot into Scala via its Scala development team. The problem is that their current approach to re-creating Scala’s type system leads to massive waste, and it doesn’t work and can’t ever work. I would like to see JetBrains invest in the core compiler to make it better suited for use in an IDE (e.g. error recovery, plug-ins to export compile-time information, tooling around TASTY, etc.). I know many Scala developers who would easily pay $500 or more for a Scala IDE that actually worked.</li>
<li><strong>Lightbend</strong>. Lightbend maintains the Scala 2.x compiler, allegedly with revenue from fintech-based support and maintenance contracts. This is important work, as Scala 3 adoption is still low in industry. However, given Lightbend has an uncertain future, and an increasingly tenuous connection to Scala, I believe that the company will stop funding 2.x maintenance. I hope the maintenance contracts will be smoothly transitioned to another organization that can continue to maintain Scala 2.x.</li>
<li><strong>LunaTech</strong>. LunaTech has not been very active in contributing to Scala, but recently the organization lent Chris Kipp to the Scala Center to focus on improved tooling for the Scala ecosystem (donating labor is one way to obtain a seat on the Scala Center advisory board). My hope is that this represents a long-term commitment from LunaTech to improving Scala tooling, and I would love to see them extend this investment.</li>
<li><strong>Scalac</strong>. Scalac is one of the oldest Scala consultancies, and has invested into many open source libraries, particularly around the Akka and ZIO ecosystems. It would be fantastic to see this investment become more consistent, dedicated, and expanded.</li>
<li><strong>SoftwareMill</strong>. SoftwareMill primarily contributes to the Scala open source ecosystem, principally its own projects such as Tapir, sttp, Bootzooka, Magnolia, MacWire, and so forth. These libraries have a positive impact on Scala’s adoption in industry. It would be great to see these OSS investments maintained, and perhaps even supplemented with additional investment into tooling or the language itself.</li>
<li><strong>VirtusLab</strong>. Virtus Lab, a consultancy whose primary business line does not involve Scala, generously sponsors some excellent open source work on Scala 3, Metals, and Scala CLI. Given that <a href="https://www.aquiline.com/">Aquiline</a>, a noted private equity firm, acquired a stake in the company and is grooming them for <del>sale to potential buyers</del> IPO, it’s unclear if this investment will continue indefinitely, but for now, we can be grateful for the work they are sponsoring.</li>
<li><strong>Xebia Functional</strong>. This is a division of Xebia formed through the acquisition of a company called 47 Degrees. As 47 Degrees, the team once contributed significantly to Scala open source (FreeStyle, Mu, Fetch, etc.). In modern times, the company’s focus has shifted to Kotlin and its Arrow OSS ecosystem. Recently the company <a href="https://contributors.scala-lang.org/t/pre-sip-suspended-functions-and-continuations/5801">controversially</a> proposed to copy Kotlin’s pre-Loom hack for the JVM’s lack of virtual threads <a href="https://github.com/47deg/TBD">into Scala</a>. Given Xebia has no vested interest in Scala, and given its resources have shifted to Kotlin, it seems unlikely the company will contribute positively to Scala.</li>
<li><strong>Ziverge</strong>. My own company Ziverge contributes mostly to the open source ZIO ecosystem, but also does a large amount of Scala evangelism through annual conferences and hackathons, and weekly educational episodes on YouTube. Ziverge is exclusively focused on Scala, and has a very clear alignment with the commercial success of Scala. I would like to see Ziverge contribute to improved tooling and cross-platform support.</li>
</ul>
<p>Given the limited resources of LAMP and the Scala Center, some of these organizations (in addition to many individual contributors) must be a part of the solution to challenges facing the Scala programming language.</p>
<h2 id="summary">Summary</h2>
<p>As a Scala enthusiast and investor, I am keen on increasing Scala’s adoption in industry. While Scala has powerful features, it also has some challenges, which prevent Scala from reaching its commercial potential.</p>
<p>I believe that it is time for all of us Scala enthusiasts to take action to improve the following areas:</p>
<ul>
<li><strong>Industry Language</strong>. Scala needs to become an industry-focused language whose innovation is powered by academia, but tightly and exclusively focused on industry. As part of this transformation, Scala 3 needs to commit to backward compatibility, like Java, Go, and all other commercial programming languages. There can never again be a “Scala 3”-style boil-the-ocean, greenfield language rewrite. It’s time for the language to stabilize, and for the focus to shift to commercial interests, including bug fixes, improved performance, enhanced tool support, and optimizing.</li>
<li><strong>World-Class Tooling</strong>. Scala needs significant investment in tooling, most notably, a commercial-grade IDE with a sustainable architecture, and a modern build tool. Beyond the IDE and build tool issues, Maven package management is antiquated and poorly suited for Scala, and Scala’s built-in REPL pales in comparison to third-party REPL support.</li>
<li><strong>Healthy Ecosystem</strong>. Scala’s open source ecosystem is good, but small, and given limited open source resources, we need to find ways to work together. Getting talented folks across ecosystems in the same room is a good first start.</li>
<li><strong>Natively Cross-Platform</strong>. Scala owes a debt of gratitude to the JVM for breathing life into this functional programming language. But ultimately, the JVM is Oracle’s baby, and tying the fate of Scala to the fate of the JVM is risky…riskier now in an age that demands applications that are lightweight, with low-memory usage, and fast startup times. Scala must go beyond the JVM in a first-class way.</li>
</ul>
<p>Given that Scala is not a commercial project, addressing these deficiencies will require coordinated effort across individual open source contributors, LAMP, Scala Center, and organizations like Ziverge. It may require organizational and operational improvements, and will most certainly require new levels of collaboration, as a diverse set of commercial and academic users come together for a common purpose.</p>
<p>Through discipline, hard work, and creative thinking, I believe we can create a new era in Scala’s commercial adoption—a true <em>Scala resurrection</em>.</p>
<p>Stay tuned for more!</p>
<p><a href="https://degoes.net/articles/scala-resurrection">Scala Resurrection</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on January 24, 2023.</p>https://degoes.net/articles/zio-config2022-11-18T00:00:00-07:002022-11-18T00:00:00-07:00John A De Goeshttps://degoes.netjohn@degoes.net<p>In a first for functional effect systems, <a href="https://degoes.net/articles/zio-2.0">ZIO 2.0</a> brought integrated metrics and logging front-ends directly into the core, which enables application developers to easily track and log what’s going on inside their applications for enhanced observability.</p>
<p>Now, some months downstream of this seminal release, we’ve witnessed rapid adoption of these features by ZIO ecosystem projects, as well as by end-users.</p>
<p>Two important factors are driving this adoption:</p>
<ol>
<li><strong>Simple, Value-Added Front-ends</strong>. Integrated logging and metrics have very simple, lightweight interfaces that impose zero barrier to adoption, and they bring value-added features powered directly by ZIO’s runtime system. For example, logging is enriched by the tracking of full failures (including concurrent failures and finalizer failures), as well as fiber identity and fork location—rich data that is freely accessible to the ZIO runtime system, and useful for monitoring operations and diagnosing production issues.</li>
<li><strong>Highly Customizable Back-ends</strong>. With the help of separate ecosystem projects <a href="http://github.com/zio/zio-logging">ZIO Logging</a> and <a href="http://github.com/zio/zio-metrics-connectors">ZIO Metrics Connectors</a>, developers can easily and rapidly wire-up ZIO’s logging and metrics front-ends to major backends, such as Logback for logging, or Prometheus for metrics monitoring. This rich integration lets you adopt ZIO’s front-end without needing to change any of the services you already use.</li>
</ol>
<p>With integrated logging and metrics, as well as overhauled dependency injection with automatic wire-up and simplified layers, ZIO 2.0 took a major leap from being a <em>Better Future</em> library to being a <em>powerful framework for modern app development</em>, powered by Scala’s unique fusion of object-oriented and statically-typed functional programming.</p>
<p>Frameworks like <a href="https://spring.io">Spring</a>, however, have long gone beyond just <em>logging</em>, <em>metrics</em>, and <em>dependency injection</em>: they also provide support for <em>configuration management</em>, which has increasingly become a pain point in the ZIO ecosystem.</p>
<h2 id="the-pain-of-config">The Pain of Config</h2>
<p>In the ZIO 2.0 service pattern, we architect our application in <em>layers</em>, each composed of <em>services</em>. Services are defined with <em>traits</em>, and are implemented with <em>classes</em>.</p>
<p>A service implementation expresses its dependency on other services via its constructor, which is lifted into a <em>layer</em>, which is capable of describing initialization, finalization, creation of state, and so forth, and which benefits from ZIO’s compile-time auto-wiring of the application dependency graph.</p>
<p>In ZIO 2.0, all layers are defined along the following lines:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">myLayer</span><span class="k">:</span> <span class="kt">ZLayer</span><span class="o">[</span><span class="kt">PaymentRepo</span>, <span class="kt">Nothing</span>, <span class="kt">MyService</span><span class="o">]</span> <span class="k">=</span>
<span class="nv">ZLayer</span><span class="o">.</span><span class="py">scoped</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">repo</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">service</span><span class="o">[</span><span class="kt">PaymentRepo</span><span class="o">]</span>
<span class="n">ref</span> <span class="k"><-</span> <span class="nv">Ref</span><span class="o">.</span><span class="py">make</span><span class="o">(</span><span class="nv">MyState</span><span class="o">.</span><span class="py">Initial</span><span class="o">)</span>
<span class="n">impl</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">succeed</span><span class="o">(</span><span class="k">new</span> <span class="nc">MyServiceImpl</span><span class="o">(</span><span class="n">ref</span><span class="o">,</span> <span class="n">repo</span><span class="o">))</span>
<span class="k">_</span> <span class="k"><-</span> <span class="nv">impl</span><span class="o">.</span><span class="py">initialize</span>
<span class="k">_</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">addFinalizer</span><span class="o">(</span><span class="nv">impl</span><span class="o">.</span><span class="py">destroy</span><span class="o">)</span>
<span class="o">}</span> <span class="k">yield</span> <span class="n">impl</span>
<span class="o">}</span></code></pre></figure>
<p>This example layer allocates resources, and registers a finalizer to dispose of them properly when the layer is being shutdown at the end of its life. For the simpler case when a service implementation does not use any resources, the <code class="language-plaintext highlighter-rouge">ZLayer.apply</code> constructor can be used.</p>
<p>It turns out that many, <em>many</em> service implementations require configuration; a requirement that I will hereafter refer to as <em>the configuration problem</em>.</p>
<p>For example, a web service implementation may need a token to call an API; a server may need a port number and certificate details; a user repo may need database and table names; and so forth.</p>
<p>This is so pervasive a pattern, that the ZIO ecosystem has developed <em>two</em> separate ways of handling it.</p>
<h3 id="solution-1">Solution #1</h3>
<p>In this solution, we solve the configuration problem by turning our layer from an ordinary value into a function, which accepts configuration, and returns the layer (a <em>layer factory</em> of sorts):</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">makeMyLayer</span><span class="o">(</span><span class="n">config</span><span class="k">:</span> <span class="kt">MyServiceImpl.Config</span><span class="o">)</span> <span class="k">=</span>
<span class="nv">ZLayer</span><span class="o">.</span><span class="py">scoped</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">repo</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">service</span><span class="o">[</span><span class="kt">PaymentRepo</span><span class="o">]</span>
<span class="n">ref</span> <span class="k"><-</span> <span class="nv">Ref</span><span class="o">.</span><span class="py">make</span><span class="o">(</span><span class="nv">MyState</span><span class="o">.</span><span class="py">Initial</span><span class="o">)</span>
<span class="n">impl</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">succeed</span><span class="o">(</span><span class="k">new</span> <span class="nc">MyServiceImpl</span><span class="o">(</span><span class="n">config</span><span class="o">,</span> <span class="n">ref</span><span class="o">,</span> <span class="n">repo</span><span class="o">))</span>
<span class="k">_</span> <span class="k"><-</span> <span class="nv">impl</span><span class="o">.</span><span class="py">initialize</span>
<span class="k">_</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">addFinalizer</span><span class="o">(</span><span class="nv">impl</span><span class="o">.</span><span class="py">destroy</span><span class="o">)</span>
<span class="o">}</span> <span class="k">yield</span> <span class="n">impl</span>
<span class="o">}</span></code></pre></figure>
<p>This solution is used probbaly 70% of the time (in my estimation), because it’s very straightforward.</p>
<p>Unfortunately, it leads to some amount of awkwardness when wiring up application dependencies, because configuration needs to be loaded <em>before</em> constructing layers:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">myApp</span> <span class="k">=</span> <span class="o">...</span>
<span class="k">def</span> <span class="nf">loadConfigs</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[(</span><span class="kt">MyConfig</span>, <span class="kt">MyOtherConfig</span><span class="o">)]</span> <span class="k">=</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">myConfig</span> <span class="k"><-</span> <span class="nf">loadMyConfig</span><span class="o">(</span><span class="n">args</span><span class="o">)</span>
<span class="n">myOtherConfig</span> <span class="k"><-</span> <span class="nf">loadOtherConfig</span><span class="o">(</span><span class="n">args</span><span class="o">)</span>
<span class="o">}</span> <span class="nf">yield</span> <span class="o">(</span><span class="n">myConfig</span><span class="o">,</span> <span class="n">myOtherConfig</span><span class="o">)</span>
<span class="k">def</span> <span class="nf">wireUp</span> <span class="k">=</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">configs</span> <span class="k"><-</span> <span class="n">loadConfigs</span>
<span class="n">myLayer</span> <span class="k"><-</span> <span class="nf">makeMyLayer</span><span class="o">(</span><span class="nv">configs</span><span class="o">.</span><span class="py">_1</span><span class="o">)</span>
<span class="o">...</span>
<span class="o">}</span> <span class="k">yield</span> <span class="nv">myApp</span><span class="o">.</span><span class="py">provide</span><span class="o">(</span><span class="n">myLayer</span><span class="o">,</span> <span class="o">...)</span></code></pre></figure>
<p>This two-stage application wireup process has a highly irregular structure that is a pain to teach, and a pain to maintain, and raises the question: why is this so difficult?</p>
<h3 id="solution-2">Solution #2</h3>
<p>In this solution, we solve the configuration problem by treating the configuration data itself as <em>another</em> service: a dependency of our service implementation.</p>
<p>Our layer definition becomes something like the following:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">myLayer</span><span class="k">:</span> <span class="kt">ZLayer</span><span class="o">[</span><span class="kt">PaymentRepo</span> <span class="kt">with</span> <span class="kt">MyServiceImpl.Config</span>, <span class="kt">Nothing</span>, <span class="kt">MyService</span><span class="o">]</span> <span class="k">=</span>
<span class="nv">ZLayer</span><span class="o">.</span><span class="py">scoped</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">repo</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">service</span><span class="o">[</span><span class="kt">PaymentRepo</span><span class="o">]</span>
<span class="n">config</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">service</span><span class="o">[</span><span class="kt">MyServiceImpl.Config</span><span class="o">]</span>
<span class="n">ref</span> <span class="k"><-</span> <span class="nv">Ref</span><span class="o">.</span><span class="py">make</span><span class="o">(</span><span class="nv">MyState</span><span class="o">.</span><span class="py">Initial</span><span class="o">)</span>
<span class="n">impl</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">succeed</span><span class="o">(</span><span class="k">new</span> <span class="nc">MyServiceImpl</span><span class="o">(</span><span class="n">config</span><span class="o">,</span> <span class="n">ref</span><span class="o">,</span> <span class="n">repo</span><span class="o">))</span>
<span class="k">_</span> <span class="k"><-</span> <span class="nv">impl</span><span class="o">.</span><span class="py">initialize</span>
<span class="k">_</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">addFinalizer</span><span class="o">(</span><span class="nv">impl</span><span class="o">.</span><span class="py">destroy</span><span class="o">)</span>
<span class="o">}</span> <span class="k">yield</span> <span class="n">impl</span>
<span class="o">}</span></code></pre></figure>
<p>This approach is more uniform (with a one-stage wire-up), but has a few drawbacks, ranging from being harder to use for simpler cases (like for tests, where we don’t need to load configuration data at all), to demanding that we have only one configuration value for each type of configuration.</p>
<p>The latter point is more subtle. Let’s say we have 10 services that each require <code class="language-plaintext highlighter-rouge">HostAndPort</code> as inputs: then when we wire up our application, they will all get the same <code class="language-plaintext highlighter-rouge">HostAndPort</code> value, rather than each getting separate <code class="language-plaintext highlighter-rouge">HostAndPort</code> configurations.</p>
<p>All these problems have workarounds, but it has felt for a long while that we can do better.</p>
<p>In ZIO 2.0.4, we are finally introducing a native configuration front-end!</p>
<h2 id="native-config-in-zio-20">Native Config in ZIO 2.0</h2>
<p>Just like integrated logging and metrics, ZIO 2.0 is introducing only a configuration <em>front-end</em>. This means that for most applications, you will need to choose a configuration <em>backend</em>, which will load configuration data and perform validation.</p>
<p>A common choice in the Scala ecosystem has been Typesafe Config, with its support for HOCON, but there are many other choices, including TOML, Yaml, JSON, XML, and of course configuration servers.</p>
<p>ZIO Config has support for some of these, but has not yet been updated to support the new configuration front-end available in ZIO 2.0.4.</p>
<p>The configuration front-end is designed to be lightweight and user-friendly, perfectly integrated with other ZIO features, and extensible via pluggable backends coming soon to <em>ZIO Config</em>.</p>
<h2 id="first-steps">First Steps</h2>
<p>To begin using ZIO 2.0 configuration, you define a <code class="language-plaintext highlighter-rouge">Config[A]</code> for some data type <code class="language-plaintext highlighter-rouge">A</code>, which holds your configuration data. A value of type <code class="language-plaintext highlighter-rouge">Config[A]</code> represents a “recipe” for decoding configuration data into a value of type <code class="language-plaintext highlighter-rouge">A</code>.</p>
<p>For example, let’s say we have the following data type:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">HostPort</span><span class="o">(</span><span class="n">host</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">port</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span></code></pre></figure>
<p>In order to use this data type for configuration with ZIO 2.0, we need to create a <code class="language-plaintext highlighter-rouge">Config[HostPort]</code>. We will recommend that users store such recipes <em>in the companion object of the data type</em>, because it’s an obvious place to look for such data.</p>
<p>To create configs, we can use the constructors in the companion object of <code class="language-plaintext highlighter-rouge">Config</code>, together with the operators in the <code class="language-plaintext highlighter-rouge">Config</code> data type itself.</p>
<p>Here’s one way to create a <code class="language-plaintext highlighter-rouge">Config[HostPort]</code>:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">HostPort</span><span class="o">(</span><span class="n">host</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">port</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span>
<span class="k">object</span> <span class="nc">HostPort</span> <span class="o">{</span>
<span class="k">val</span> <span class="nv">config</span><span class="k">:</span> <span class="kt">Config</span><span class="o">[</span><span class="kt">HostPort</span><span class="o">]</span> <span class="k">=</span>
<span class="o">(</span><span class="nv">Config</span><span class="o">.</span><span class="py">string</span><span class="o">(</span><span class="s">"host"</span><span class="o">)</span> <span class="o">++</span> <span class="nv">Config</span><span class="o">.</span><span class="py">int</span><span class="o">(</span><span class="s">"port"</span><span class="o">)).</span><span class="py">map</span> <span class="o">{</span>
<span class="nf">case</span> <span class="o">(</span><span class="n">host</span><span class="o">,</span> <span class="n">port</span><span class="o">)</span> <span class="k">=></span> <span class="nc">HostPort</span><span class="o">(</span><span class="n">host</span><span class="o">,</span> <span class="n">port</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></figure>
<p>Once we have defined a <code class="language-plaintext highlighter-rouge">Config[A]</code> for some data type <code class="language-plaintext highlighter-rouge">A</code>, we can load this configuration using <code class="language-plaintext highlighter-rouge">ZIO.config(...)</code>, which will either successfully load the data from the configuration backend, or it will produce a detailed error about what is missing or malformed.</p>
<p>Now layer construction will end up looking something like the following:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">myLayer</span><span class="k">:</span> <span class="kt">ZLayer</span><span class="o">[</span><span class="kt">PaymentRepo</span>, <span class="kt">Nothing</span>, <span class="kt">MyService</span><span class="o">]</span> <span class="k">=</span>
<span class="nv">ZLayer</span><span class="o">.</span><span class="py">scoped</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">repo</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">service</span><span class="o">[</span><span class="kt">PaymentRepo</span><span class="o">]</span>
<span class="n">config</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">config</span><span class="o">(</span><span class="nv">MyServiceImpl</span><span class="o">.</span><span class="py">config</span><span class="o">)</span>
<span class="n">ref</span> <span class="k"><-</span> <span class="nv">Ref</span><span class="o">.</span><span class="py">make</span><span class="o">(</span><span class="nv">MyState</span><span class="o">.</span><span class="py">Initial</span><span class="o">)</span>
<span class="n">impl</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">succeed</span><span class="o">(</span><span class="k">new</span> <span class="nc">MyServiceImpl</span><span class="o">(</span><span class="n">config</span><span class="o">,</span> <span class="n">ref</span><span class="o">,</span> <span class="n">repo</span><span class="o">))</span>
<span class="k">_</span> <span class="k"><-</span> <span class="nv">impl</span><span class="o">.</span><span class="py">initialize</span>
<span class="k">_</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">addFinalizer</span><span class="o">(</span><span class="nv">impl</span><span class="o">.</span><span class="py">destroy</span><span class="o">)</span>
<span class="o">}</span> <span class="k">yield</span> <span class="n">impl</span>
<span class="o">}</span></code></pre></figure>
<p>Notice how this layer is just an <em>ordinary value</em>, not a method, and that configuration does not appear <em>anywhere</em> inside the dependencies of the service (the input to the layer).</p>
<p>How is this configuration data loaded?</p>
<p>Through a new service in ZIO 2.0 called <code class="language-plaintext highlighter-rouge">ConfigProvider</code>. A <code class="language-plaintext highlighter-rouge">ConfigProvider</code> is capable of loading configuration data from any <code class="language-plaintext highlighter-rouge">Config</code>. Configuration providers can be composed using fallback semantics, as well as transformed in various ways.</p>
<p>By default, ZIO 2.0 ships with a <code class="language-plaintext highlighter-rouge">ConfigProvider</code> that looks <em>first</em> at environment variables, and <em>second</em> at system properties. This <code class="language-plaintext highlighter-rouge">ConfigProvider</code> can be used to bootstrap other, more sophisticated configuration providers, which may need to read files mounted on networked drives or inside an image, or possibly connect to configuration servers.</p>
<p><code class="language-plaintext highlighter-rouge">ConfigProvider</code> is stored in a fiber ref, so it can be updated locally in parts of your application, and that’s exactly how an upcoming version of <em>ZIO Config</em> will tie different backends into the front-end.</p>
<h2 id="disambiguation">Disambiguation</h2>
<p>In order to disambiguate different instances of the same configuration data, <code class="language-plaintext highlighter-rouge">Config</code> provides a method called <code class="language-plaintext highlighter-rouge">Config#nested</code>, which allows you to provide a namespace for configuration data.</p>
<p>In the preceding example, we might actually load the configuration using <code class="language-plaintext highlighter-rouge">ZIO.config(HostPort.config.nested("MyService"))</code>. This would allow us to configure <code class="language-plaintext highlighter-rouge">MyService</code> separately from other parts of the application that also require <code class="language-plaintext highlighter-rouge">HostPort</code>:</p>
<pre>
[MyService]
host = localhost
port = 8080
</pre>
<p>The <code class="language-plaintext highlighter-rouge">nested</code> method is currently the recommended way of achieving disambiguation, although future versions of ZIO may be able to take advantage of the application dependency graph.</p>
<h2 id="summary">Summary</h2>
<p>As you can see, the new configuration front-end in ZIO 2.0.4 is simple and cleanly integrated into ZIO 2.0, with powerful flexibility that enables us to reuse our existing application configuration.</p>
<p>Over the course of the next few months, core ZIO libraries like ZIO HTTP, ZIO JDBC, ZIO Redis, and others, will adopt ZIO 2 configuration so that you have a standardized, uniform way of configuring different components across the entire ZIO ecosystem. Meanwhile, <em>ZIO Config</em> will be updated to provide seamless support for both traditional configuration backends (files and the like), as well as more modern configuration backends, such as Docker, Kubernetes, and cloud configuration services.</p>
<p>With this step of integrated configuration, ZIO 2 continues down the path of solving all the common painpoints of cloud application development, with the principled, type-safe, and user-friendly approach that ZIO has pioneered in the space of object-oriented, pure functional programming.</p>
<p>Please give the new feature a whirl, and let us know what you think!</p>
<p><a href="https://degoes.net/articles/zio-config">Native Config Lands in ZIO 2.0 </a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on November 18, 2022.</p>https://degoes.net/articles/zio-2.02022-06-24T00:00:00-06:002022-06-24T00:00:00-06:00John A De Goeshttps://degoes.netjohn@degoes.net<p>On August 3rd, 2020, the ZIO community <a href="https://degoes.net/articles/zio-1.0">released ZIO 1</a>—itself the product of 3 years of intensive research & development and continuous refinement based on commercial adoption and early production feedback.</p>
<p>The initial release of ZIO exceeded expectations, introducing <a href="https://github.com/zio/zio">dozens and dozens</a> of name-brand companies to the power of functional Scala. As ZIO had done since it was called the “Scalaz 8 IO monad”, ZIO 1 pushed forward the state-of-the-art for effect systems, with async stack tracing, fiber dumps, and other novel features pioneered by the ZIO community.</p>
<p>To this day, the 1.0 release of ZIO has held up brilliantly, still offering the most expressive, type-safe, and feature-packed effect system in Scala, boasting features that are years away in other effect systems, including automatic structured concurrency, typed errors, context, schedules, and software transactional memory.</p>
<p>Yet, only dead software doesn’t evolve. For ZIO to remain relevant to the future of functional Scala, ZIO 1 <em>had</em> to innovate. So, for the past 2 years, adopters of ZIO have been sharing feedback. Some positive, some negative, and all of it incredibly useful.</p>
<p>The result of this feedback, combined with keen insight, research and development, and innovation from the core ZIO team (including OSS superstars <em>Adam Fraser</em>, <em>Daniel Vigovszky</em>, and <em>Kit Langton</em>), is the ZIO 2 release.</p>
<p>Officially previewed at ZIO World 2021, and tested in multiple release candidates over many months now, ZIO 2 represents a <em>giant</em> leap forward for functional effect systems in Scala.</p>
<p>Today, after more than a year of eager anticipation from ZIO users worldwide, I am extremely excited to announce the official release of ZIO 2.0!</p>
<h2 id="the-big-picture">The Big Picture</h2>
<p>Throughout the ZIO 2 release, you will find five major themes organzing all the features and improvements that have been made to the library:</p>
<ul>
<li><strong>Runtime System</strong>. ZIO 2 pushes forward the state-of-the-art with a novel, third-generation runtime system that’s been optimized for Loom, and which, even in its early state, shows promise to be the fastest non-macro-based runtime system.</li>
<li><strong>Developer Experience</strong>. ZIO 2 tries to eliminate many of the rough edges in ZIO 1, improve consistency, simplify the number and surface area of data types, and adopt architectural patterns that are more familiar to Java developers.</li>
<li><strong>Monitoring</strong>. ZIO 2 improves and adds a whole host of new functionality designed to make it easier to monitor and troubleshoot ZIO 2 applications deployed in production. Much of this functionality will seamlessly benefit other libraries in the ZIO ecosystem.</li>
<li><strong>Streaming</strong>. ZIO 2 completely re-imagines the foundations of streaming by unifying streams, sinks, and transformers through channels, providing a single, highly-expressive, principled way of solving data engineering challenges.</li>
<li><strong>Performance</strong>. ZIO 2 retains its focus on high-performance commercial functional programming, achieving some significant improvements. Although there is more work to be done, particularly on the new runtime system and streams, ZIO 2 is very fast.</li>
</ul>
<p>While it’s impossible to cover the scope of all of these improvements in a single post, I’m going to give you a tour of the highlights for each area, beginning with the one that’s sure to be the biggest surprise to ZIO users: the <em>brand-new</em> runtime system!</p>
<h2 id="runtime-system">Runtime System</h2>
<p>The runtime system is the <em>engine</em> that powers ZIO, executing ZIO effects in a way that preserves all the guarantees that ZIO provides around efficiency, resource-safety, concurrency, error handling, and asynchronicity.</p>
<p>The runtime system is the single most important component of any functional effect system: it must be bulletproof, performant, and tractable for reasons of maintenance, testing, and auditing.</p>
<p>ZIO World 2021 previewed some improvements to the ZIO runtime system, but these improvements have now been <em>obsoleted</em> by a <strong>complete rewrite of the ZIO runtime system</strong>.</p>
<p>ZIO 2 now features the world’s first <strong>third-generation engine</strong> for any Scala effect system, completed, tested, and unveiled <em>just in time</em> for the release of ZIO 2 final.</p>
<h3 id="a-brief-history-of-engines">A Brief History of Engines</h3>
<p>The <strong>first-generation engines</strong> included Scalaz Task and initial versions of Monix. They utilized an advanced technique called <em>trampolining</em> to achieve stack safety and support asynchronous execution without direct runtime support. Despite their power and benefits, they suffered from immature cancellation models, and they had no concept of fibers—they lacked pure forking and pure cancellation, and they did not have a well-developed model for resource safety in the presence of automatic cancellation.</p>
<p>A pre-release version of ZIO unveiled the first <strong>second-generation engine</strong>, with a clear fiber-based execution model, pure forking and cancellation, and integrated, cancellation-aware resource safety. Cats Effect 1.0 underwent an <em>eleventh hour makeover</em> to support these second-generation features (Cats Effect 3.x replicates more ZIO 1 features but does not change the core design).</p>
<p>For a long while, I did not think there <em>would</em> be a third-generation engine, as the fundamental approach to designing runtimes for effect systems has remained unchanged for years.</p>
<p>In working on ZIO 2, however, I was determined to improve <em>integration performance</em>. We know from experience that Java libraries (and some Scala libraries) frequently need to call into ZIO applications, and execute their effects. For example, a Java library that places an element into a ZIO queue needs to execute the effect returned by <code class="language-plaintext highlighter-rouge">Queue.offer</code>.</p>
<p>My interest in accelerating these integration use cases led me to develop a so-called <em>fast-path interpreter</em>, designed specifically to accelerate these (predominantly synchronous) use cases.</p>
<h3 id="fast-path">Fast-Path</h3>
<p>The fast-path interpreter I designed for ZIO 2 eschewed traditional effect system architecture. For one, it did not use a trampoline, which meant it could not handle asynchronous effects.</p>
<p>Executing effects directly on the JVM stack, and only supporting a few of the ZIO operations, the fast-path interpreter tried to make as much progress as possible on its own, before ultimately giving up and delegating to the normal ZIO runtime system.</p>
<p>The fast-path interpreter was able to completely execute integration effects (like adding an element to a queue). This provided a significant performance boost in realistic scenarios, up to 100x faster than Cats Effect 3.x, and several times faster than ZIO 1.x.</p>
<p>I was pleased by these results, and I intended the fast-path interpreter to be a defining feature of ZIO 2, offering best-in-class performance for interop use cases. But I was also disappointed at having to maintain <em>two</em> runtime systems, and deep down, I felt like ZIO’s main runtime system could incorporate some learnings from the fast-path interpreter.</p>
<p>The only problem is, I didn’t know how, exactly.</p>
<h3 id="asynchronous-hell">Asynchronous Hell</h3>
<p>The primary reason the fast-path interpreter couldn’t be used as the main runtime system is the existence of asynchronous operations. When a fiber executes an asynchronous operation, it must be suspended, with further instructions stored on the heap in a so-called <em>reified stack</em> (a todo list, of sorts, keeping track of what we are supposed to do <em>after</em> the async operation completes).</p>
<p>By the first quarter of 2022, ZIO 2 was almost complete, having just eliminated <em>ZManaged</em>, in favor of a radically simpler, faster, and more powerful design using <em>scopes</em>.</p>
<p>In theory, we could ship ZIO 2 whenever we wanted, after we polished documentation and interfaces. Yet, I couldn’t resist the urge to experiment with a different kind of engine, based on my experience designing and optimizing the fast-path interpreter.</p>
<h3 id="a-breakthrough">A Breakthrough</h3>
<p>On a long and grueling trans-Atlantic trip in April, I achieved a milestone: I prototyped a hybrid engine that tried to execute effects directly on the JVM stack, but which could also handle async operations by carefully and gradually unwinding the JVM stack, saving instructions for further processing on the heap in a reified stack.</p>
<p>In purely synchronous code, this prototype interpreter demonstrated faster performance than every second-generation engine, including ZIO 1 and Cats Effect 3.x. Unlike the fast-path interpreter, however, the prototype could also handle asynchronous operations. I quickly generalized the prototype to handle infinite tail-recursion using the same technique.</p>
<p>This crude prototype, however promising it might have been, was a world away from being a <em>production-grade</em> ZIO runtime system. Production runtime systems have to handle interruptibility (precise regions in which a fiber may or may not cancel an executing effect), finalization (executing cleanup actions during errors or interruption), concurrency, error reporting, error accumulation, stack trace generation (an obstacle as big as async!), and so much more.</p>
<p>There was no possible way to simply “graft” the prototype into the ZIO runtime system. The design was radically different and totally incompatible. So I had a choice: push the new design as far as possible, to see if it could become more than a toy; or abandon the prototype altogether.</p>
<p>As you can probably tell by the delayed release date for ZIO 2, I chose the former option!</p>
<h3 id="blank-sheet">Blank Sheet</h3>
<p>Though daunted by the prospect of rewriting the ZIO runtime system (and potentially blocking the release of ZIO 2 by weeks or even months), I was also intrigued by the possibility of rethinking some fundamental design decisions made in the ZIO 1 runtime system.</p>
<p>ZIO 1 used immutable data to represent large parts of the fiber state, which allowed use of atomic references to coordinate changes across disparate parts of the fiber state. Unfortunately, the use of immutable data also led to increased heap churn and GC.</p>
<p>ZIO 1 also touched the heap more often than theoretically necessary in the hot path, accessing no less than two volatile variables for each iteration of its run loop (this grew to three volatile variables in the series/2.x branch, when fiber dumps were turned on by default).</p>
<p>If I was starting from a blank slate, I wanted the new runtime system to be able to use mutable data, and to minimize accessing the heap (especially volatile variables).</p>
<h3 id="the-new-design">The New Design</h3>
<p>After an intense series of weeks in which I spent nights and weekends writing and rewriting the new runtime system, and with the help of my colleague Adam Fraser at <em>ZIO Hackathon Scotland</em>, I came up with a design for the ZIO 2 runtime system that achieved all of my goals.</p>
<p>In the coming weeks, I will give a talk about the new design, but for purposes of the ZIO 2 release, it’s enough for me to highlight the achievements of this design, which include:</p>
<ul>
<li><strong>Microkernel</strong>. ZIO 1 achieved a remarkably small kernel by expressing parts of the ZIO execution semantics in ZIO itself. The ZIO 2 runtime system takes this technique and augments it with a hybrid declarative/executable encoding that reduces the size of the microkernel by 30%. The runtime system is leaner and meaner than any effect system in history.</li>
<li><strong>Trampoline Avoidance</strong>. The ZIO 2 runtime system attempts to fully execute effects without a trampoline, and will only fallback to a reified stack in the presence of an asynchronous operation, an overly deep call stack, or a fiber that is hogging execution time. In Loom, there is never a need for asynchronous operations, which makes the ZIO 2 runtime system highly optimized for Loom.</li>
<li><strong>Mutable State</strong>. The ZIO 2 runtime system utilizes a nano-actor core, which allows it to provide concurrent-safe implementation in the presence of fully mutable state, decreasing allocations and GC pressure. I have been and remain a vocal critic of actors in user-land code, but I like to think that use of actors in ZIO 2 is vindication that the actor model is useful in the design of concurrent and distributed systems (albeit as a private implementation detail).</li>
<li><strong>Local State</strong>. The ZIO 2 runtime system stores all runtime flags (including interruptibility), as well as the line of code it is currently executing, on the JVM stack (not the heap), and as a result, only needs to touch the heap to process new messages in its message queue, which are infrequent and arise primarily from fiber-to-fiber interaction.</li>
<li><strong>Cost-Free Tracing</strong>. The ZIO 2 runtime system achieves truly cost-free async stack traces for the happy path (when there are no errors). Given the 1.x runtime imposed 2-3x overhead on the happy path, this is a phenomenal achievement that will make a noticeable improvement in the behavior of real world ZIO applications, which have almost universally been deployed with tracing enabled.</li>
</ul>
<p>The end result of this design is the world’s first <em>third-generation engine</em> for functional effect systems. The design achieves all the benefits of second-generation engines, but with an extreme amount of happy-path specialization that obviates the need for trampolining in many cases, and a super clean and minimal design that provides some assurance of correctness and stability.</p>
<p>The new runtime system has proven so solid, we’ve enabled previously ignored tests and added new test suites to give more rigorous semantics to interruption behavior.</p>
<h3 id="benefits">Benefits</h3>
<p>As of the date of this writing, we are just weeks away from the moment when the new runtime system first successfully passed the battery of automated tests that helps verify runtime system integrity.</p>
<p>As a result of the compressed time frame and our intense focus on correctness and stability, we are still early in the process of tweaking, tuning, and optimizing the new runtime system.</p>
<p>That said, the benchmarks point to one undeniable result: the ZIO 2 runtime system is <em>very fast</em> at synchronous workloads.</p>
<p>The following benchmark compares ZIO 2 with both ZIO 1 and Cats Effect 3 on a fibonacci example that is frequently used to benchmark functional effect systems:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Benchmark (depth) Mode Cnt Score Error Units
ZIO 1.x 20 thrpt 20 647.268 ± 12.892 ops/s
Cats Effect 3.x 20 thrpt 20 947.935 ± 124.824 ops/s
ZIO 2 20 thrpt 20 1903.838 ± 21.224 ops/s
</code></pre></div></div>
<p>In fairness, and in the spirit of full transparency, I will note that ZIO 2 is currently <em>slower</em> than both ZIO 1 and Cats Effect 3 on async heavy workloads (workloads that always require trampolining), which was expected given the costs of lazy trampolining.</p>
<p>However, we are confident enough in our plan for improving async performance and in the importance of optimizing for Loom that we believe the ZIO 2 runtime system sets a new standard for innovation in functional effect systems.</p>
<p>In the coming weeks following the release, we will push forward the performance of the new runtime system, and publish a full analysis of its capabilities, with a special emphasis on what Loom-powered applications can expect to achieve under ZIO 2.</p>
<h2 id="developer-experience">Developer Experience</h2>
<p>Since ZIO 1 launched, we’ve seen new contributors like Kit Langton join the project with fresh ideas and a heightened passion for making ZIO a joy to use for beginners and experts alike.</p>
<p>We’ve also had a chance to see how our previous attention to developer experience resulted in significant adoption and happy users, and I think this motivated all contributors (including myself) to obsess over how we can make the experience of using ZIO 2 delightful, unsurprising, and familiar.</p>
<p>In the following sections, I’ll summarize some of the key improvements to developer experience.</p>
<h3 id="environment">Environment</h3>
<p>In ZIO 1, the environment often ended up looking like <code class="language-plaintext highlighter-rouge">Has[UserRepo] with Has[ConnectionPool]</code> (Has-spaghetti!), and the environment APIs were confusing, because some methods expected you to wrap your services in <code class="language-plaintext highlighter-rouge">Has</code>, and others expected you to <em>not</em> wrap your services in <code class="language-plaintext highlighter-rouge">Has</code>.</p>
<p>In ZIO 2, the <code class="language-plaintext highlighter-rouge">Has</code> data type has been refactored into a more straightforward data type called <code class="language-plaintext highlighter-rouge">ZEnvironment</code>, which uses a phantom type parameter to track its contents. <code class="language-plaintext highlighter-rouge">ZEnvironment</code> itself has been baked into ZIO, so most users never need to know that it exists.</p>
<p>As a result, ZIO environments now have a super clean appearance and much cleaner API:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">effect</span> <span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">UserRepo</span> <span class="kt">with</span> <span class="kt">ConnectionPool</span>, <span class="kt">Throwable</span>, <span class="kt">Unit</span><span class="o">]</span></code></pre></figure>
<p>Concurrent with this change, ZIO environment now embraces subtyping, which means that if you stick a <code class="language-plaintext highlighter-rouge">Dog</code> in the environment, you can now get both a <code class="language-plaintext highlighter-rouge">Dog</code> and an <code class="language-plaintext highlighter-rouge">Animal</code> out of the environment. Previously, if you stuck a <code class="language-plaintext highlighter-rouge">Dog</code> in the environment, you could only get a <code class="language-plaintext highlighter-rouge">Dog</code> out of the environment (even when <code class="language-plaintext highlighter-rouge">Dog</code> was a subtype of <code class="language-plaintext highlighter-rouge">Animal</code>), due to the invariance of <code class="language-plaintext highlighter-rouge">Has</code>.</p>
<p>To make this interface sound, ZIO 2 uses metaprogramming to prevent you from retrieving intersections from the environment (you can safely ignore this if you don’t know type theory!).</p>
<h3 id="layers">Layers</h3>
<p>In ZIO 2, Kit Langton redesigned layers to tackle one of the biggest pain points of ZIO 1. With the aid of metaprogramming, ZIO 2 wires your layers together automatically.</p>
<p>This type-safe dependency injection is capable of wiring up an entire application dependency graph, parallelizing construction to minimize startup times, and appropriately handling asynchronous initialization, initialization failures, and resource-safety.</p>
<p>If you fail to specify all layers required by an effect, ZIO 2 will give you detailed, rich error messages, which guide you to satisfying all requirements of your effect.</p>
<p>ZIO 2 also deletes almost all ways to create layers, instead favoring very simple and predictable patterns of layer construction that can be used for simple or complex services.</p>
<h3 id="auto-blocking">Auto-Blocking</h3>
<p>ZIO 1 was the first effect system to introduce the concept of a dedicated blocking pool, and operators to shift effects to run on this pool. While controversial, eventually other Scala effect systems replicated this behavior, and now the feature is standard.</p>
<p>The separate blocking pool allows you to make sure that blocking operations do not accidentally use up the worker threads from ZIO’s main thread pool, allowing you to achieve higher application throughput from the resulting increased efficiency.</p>
<p>While important for creating high-performance applications, the need to manually shift blocking workloads to the blocking thread pool has proven a usability hurdle, because users often are not aware of which Java methods block threads, and which do not. If they make mistakes, then application throughput suffers, without users being aware of what’s going on.</p>
<p>In ZIO 2, the new fiber-aware scheduler designed by Adam Fraser will monitor the execution of fibers and intelligently learn which fibers engage in blocking behavior. The scheduler shifts these blocking fibers automatically to the blocking thread pool. This innovative, smart behavior enables users to achieve great throughput while still focusing on business logic, without having to know exactly which methods may end up executing blocking IO operations.</p>
<h3 id="consistency">Consistency</h3>
<p>In ZIO 2, we have gone through every data type and every API, trying to make naming and organization as consistent as possible, so that your knowledge of one part of ZIO will transfer to other parts of ZIO. The greater consistency will teach your brain to see patterns between different methods and different data types—patterns that were there before, but which were obfuscated by ad hoc naming differences.</p>
<h3 id="resources">Resources</h3>
<p>In ZIO 2, Adam Fraser successfully deleted the <code class="language-plaintext highlighter-rouge">ZManaged</code> data type (or, more precisely, moved it out of core and into a legacy package), because we found that ZIO environment was powerful enough to provide the same resource-safety guarantees, but with just ZIO.</p>
<p>Using a new concept called <em>scopes</em>, ZIO effects can now easily indicate in their types whether they leave resources open that need to be closed later, and a few simple operators allow closing scopes to ensure that all open resources are safely released.</p>
<p>In addition to improving performance, this change makes it easier for beginners to start writing resource-safe code, and it also improves correctness because there are no duplicated implementations for concurrent methods, which there were with <code class="language-plaintext highlighter-rouge">ZManaged</code>. Finally, the change improves expressiveness, because ZIO has many more methods than <code class="language-plaintext highlighter-rouge">ZManaged</code>, and now you can use all of these methods directly with resources.</p>
<h3 id="architecture">Architecture</h3>
<p>In ZIO 2, we made a conscious decision to embrace object-oriented architecture, where you define components of your application using interfaces, and you implement them with classes, which use their constructors to express their requirements on other interfaces.</p>
<p>This shift toward module-oriented dependency management is familiar to Java programmers, eliminates the need for accessors, and frees up ZIO environment for transient contextual information, or plumbing dependencies through library code.</p>
<h2 id="monitoring">Monitoring</h2>
<p>In ZIO 2, we’ve spent a lot of time thinking about what happens after you deploy your application into production. This thinking has translated into numerous improvements designed to help you (or, in some cases, your ops team) have a great experience, led by Andreas Gies, myself, and others.</p>
<h3 id="traces">Traces</h3>
<p>ZIO 1 pioneered asynchronous traces and execution tracing, inspiring Cats Effect to replicate the functionality into their own effect type.</p>
<p>While traces were a game-changer, they had a number of drawbacks, chiefly that they did not look anything like exception traces, and that they imposed a 2x or worse performance overhead on application code.</p>
<p>In ZIO 2, traces have been completely redesigned to emulate the appearance of Java stack traces, and they use a novel implementation in the runtime system that results in no performance overhead in the happy path.</p>
<p>Async traces are always turned on, and cannot be turned off, ensuring that every error which occurs in a ZIO application comes with rich context to help diagnose what went wrong.</p>
<h3 id="fiber-dumps">Fiber Dumps</h3>
<p>ZIO 1 also pioneered fiber dumps, which enable printing out a dump of fibers to see which fibers are running, which are suspended, and which are blocked by other fibers. However, fiber dumps in ZIO 1 were optional and required ZIO ZMX.</p>
<p>In ZIO 2, fiber dumps are now enabled by default, and require no configuration or libraries.</p>
<h3 id="logging">Logging</h3>
<p>The ZIO runtime system has a lot of incredibly useful information, including what fiber your code is running on, context associated with the fiber, the line of code every fiber is executing, and much more.</p>
<p>In an effort to make this rich information more available for monitoring, ZIO 2 incorporates a logging front-end, which is expected to become the default logging facade for all ZIO 2 libraries and applications.</p>
<p>In a separate project called <em>ZIO Logging</em>, users can tie ZIO 2’s logging into whatever logging backend they prefer, from Logback to Java logging and everything in between.</p>
<h3 id="metrics">Metrics</h3>
<p>ZIO 2 features integrated metrics, including the ability to create user-defined metrics out of a variety of metric types (histograms, summaries, frequencies, counters, and gauges).</p>
<p>ZIO 2’s runtime system utilizes these metrics to provide detailed real-time data on running ZIO 2 applications, including fiber life times, fiber exit failures, fiber errors, and fiber counts.</p>
<p>In a separate project called <em>ZIO Metrics Connectors</em>, users can plug ZIO 2’s metric system into whatever monitoring solution they choose, including NewRelic, DataDog, and Prometheus.</p>
<h2 id="streaming">Streaming</h2>
<p>ZIO 1’s model of streaming was based on three distinct data types: streams (which produce values), transducers (which transform values), and sinks (which consume values).</p>
<p>If you wanted something else, or perhaps a hybrid that combined aspects of these data types, you had to build your own data type. At the same time, it was quite apparent to those of us who worked on the internals of streams that these data types are related to each other.</p>
<p>Thanks to herculean efforts by Daniel Vigovszky, Itamar Ravid, and Adam Fraser, this has all changed in ZIO 2, which features a completely overhauled streaming engine.</p>
<p>We evaluated many potential foundations for streaming, before <em>channels</em> emerged as the clear winner. A channel has two ends (a bit like a real-life <em>pipe</em>), and reads information on one end, and writes it on the other. Technically, channels read both termination and elements from an upstream channel, and write termination and elements to a downstream channel. The fundamental composition operator is connecting two channels together to form a composite channel.</p>
<p>In ZIO 2, streams, pipelines (the successor for transducers), and sinks are all thin wrappers around channels, creating a common unified code base that minimizes duplication. The runtime system for channels can fully optimize whole streaming pipelines, because ultimately, a fully composed pipeline (even one that is constructed from streams, pipelines, and sinks) is still just a channel.</p>
<p>Channels provide a rigorous foundation for ZIO 2 streams. Although our work on streams is far from done (in particular, we will improve performance and compositionality in 2.1), ZIO 2 channels present a giant leap from ZIO 1 streams, and contain the most rigorously complete yet minimal foundation for streaming in the entire Scala ecosystem.</p>
<h2 id="performance">Performance</h2>
<p>The new runtime system in ZIO 2 was introduced primarily for performance reasons, and while more work remains to be done here around async workloads, initial results are amazing. ZIO 2 is a perfect fit for a post-Loom world, and its hybrid engine has set a new standard for functional effect systems.</p>
<p>Beyond the new runtime system, ZIO 2 introduces new high-performance data types like <em>Hub</em> and <em>Pool</em>, a new fiber-aware scheduler that is directly inspired by Tokio (an asynchronous runtime for Rust), much faster resource safety, faster async stack tracing, higher throughput due to auto-blocking, faster layer construction, and so much more.</p>
<h2 id="conclusion">Conclusion</h2>
<p>ZIO introduced the Scala world to the power of commercial functional programming, with an innovative feature set that inspired and set the standard for other effect systems.</p>
<p>ZIO 2 builds on this tradition by acknowledging and learning from mistakes, and innovating in numerous areas, with the result that ZIO 2 is tremendously more powerful, far easier to use, and significantly easier to operate in production.</p>
<p>If you have been skeptical of what functional programming can bring to the table for your business, then now is your chance to put functional Scala to the test.</p>
<p>You won’t be disappointed. And if you <em>are</em> disappointed, we’ll happily listen to your feedback for ZIO 3!</p>
<p><a href="https://degoes.net/articles/zio-2.0">ZIO 2.0 Released</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on June 24, 2022.</p>https://degoes.net/articles/travis-brown-abuser2021-11-19T00:00:00-07:002021-11-19T00:00:00-07:00John A De Goeshttps://degoes.netjohn@degoes.net<p>I’m a technologist and a maker. I love to build. I am not interested in corporate politics or virtue signaling.</p>
<p>More than that, I believe most drama in tech is rooted in <a href="https://hbr.org/2010/04/envy-at-work">professional jealousy</a> and <a href="https://www.semanticscholar.org/paper/Feeling-of-Security-Inse-curity-and-Prejudice%3A-A-Roy-Arora/e81602188a9acf23ff8f97bdb941171431e40b58">personal insecurity</a>.</p>
<p>I have no desire to participate in such drama. I’d rather focus on creating beautiful and amazing technology that changes the world for the better, and empowering others to do the same.</p>
<p>Yet, there comes a point when even pure technologists such as myself need to wade through the muck long enough to support our colleagues and peers.</p>
<p>For me, that time is <em>now</em>.</p>
<h2 id="travis-brown">Travis Brown</h2>
<p>An <a href="https://www.reddit.com/r/scala/comments/qweo20/a_statement_about_my_scala_open_source_worktravis/">ex-Scala developer</a> by the name of <em>Travis Brown</em> has been obsessed with me since 2016, outraged that <a href="https://degoes.net/articles/lambdaconf-inclusion">LambdaConf</a> did not cave to the demands of his own organization, <a href="https://degoes.net/articles/zio-professionalism">TypeLevel</a>.</p>
<p>As organizer of LambdaConf, I made a decision to not override the vote of our minority speakers on whether a developer named Curtis Yarvin could speak about Urbit (his open source project), despite Yarvin’s alternate life as a pseudonymous neoreactionary political blogger. His proposal had previously been accepted by the double-blind committee based on merit, without my input.</p>
<p>Now, reasonable people can and do disagree with my decision, and I respect their perspectives, even when I do not always agree with them. I have learned a lot from the ordeal, and these days, with events like <a href="https://functionalscala.com">Functional Scala</a>, I try to curate invitations to ensure these professional events remain focused solely on technology and not political drama.</p>
<p>That said, what could have been a one-time statement from Travis Brown became a never-ending and escalating dark obsession. With each new talk I gave in the Scala community, with each person I mentored, and with each open source library I contributed to, Travis Brown’s rage grew.</p>
<p>Over the course of nearly half a decade, this blind rage has crossed numerous ethical, professional, and even legal lines, including:</p>
<ul>
<li><strong>Cyberstalking</strong>. Travis Brown meticulously tracks all my public posts, likes, and shares across all social media platforms (Github, Twitter, etc.), archives them permanently (without my consent, and against the terms of service of these organizations, and against data protection laws in some countries), and then posts them online with his own grotesque and twisted interpretations.</li>
<li><strong>Career Sabotage</strong>. Travis Brown attempted to cancel my business partnerships, and tried to bully technology events to stop inviting me to give talks. Travis Brown was unsuccessful in these attempts, except for SkillsMatter, which canceled my joint keynote with Wiem Zine Elabidine, but soon imploded after a torrent of backlash against their decision.</li>
<li><strong>Online Harassment</strong>. Travis Brown directly and specifically targets me in numerous online posts, usually in an inflammatory, aggressive, and sometimes verbally abusive fashion, which results in his followers attacking me. This behavior is sometimes called “trolling”, but make no mistake, Travis Brown’s “trolling” is a form of online harassment.</li>
<li><strong>Defamation</strong>. Travis Brown has consistently and repeatedly made false statements about me with the intent to damage my reputation. I am a liberal, with a strong track record of supporting LGBT+ rights, working against prejudice, and donating significant time and money to help disadvantaged individuals break into tech. I have succeeded in creating one of the most diverse open source communities in the Scala community, and I have <a href="https://degoes.net/articles/zio-professionalism">no tolerance</a> for prejudiced behavior in spaces I managed. Despite this, Travis Brown consistently paints me as an “alt-right supporter”, or worse.</li>
</ul>
<p>None of this behavior is ever remotely acceptable. In fact, not only is it unacceptable, but it should always be condemned, in the strongest possible terms.</p>
<p>Yet, due to my reluctance to wade into drama, and due to my desire to remain completely focused on technology, I have largely ignored this abusive behavior from Travis Brown.</p>
<p>Indeed, I would have continued to ignore his behavior indefinitely.</p>
<p>As a leader in open source and the CEO of a technology company, I <em>expect</em> to be criticized for decisions that I make, and it is my responsibility to <em>hear</em> that criticism. Given my visibility, I expect not only to be criticized, but to be insulted, harassed, stalked, and even defamed.</p>
<p>However, even I have my limits, and two weeks ago, when Travis Brown started attacking Martin Odersky and personal friends of mine, it crossed a line.</p>
<p>Now is the time for me to speak up. Not in my own defense, but in defense of <em>others</em>.</p>
<h2 id="martin-odersky">Martin Odersky</h2>
<p>Creator of the <a href="https://www.scala-lang.org">Scala programming language</a>, Martin Odersky is a visionary who foresaw the rise of functional programming at a time when many of us were too obsessed with object-oriented programming to care what a ‘lambda’ was.</p>
<p>Odersky sought to create a language that would combine the large-scale modularity of object-oriented programming with the power and elegance of functional programming. With Scala, <a href="https://www.youtube.com/watch?v=NY2ZkcYZj54">he has succeeded</a>.</p>
<p>Odersky does not wade into open source wars (for example, which JSON library is better). Nor does he take sides in which style of Scala is better (although he has his own opinions). Rather, he characterizes the Scala community as a “big tent”, with room to hold everyone, even if they don’t always agree or get along with each other.</p>
<p>Odersky directs the evolution of the Scala programming language, and his decisions are sometimes heavily criticized. Yet, like any good leader, Odersky does not take criticism of Scala personally. Instead, he listens to criticism and often uses the feedback to improve the language.</p>
<p>Indeed, even when people have crossed a line on the Scala Contributors forum, and personally attacked Odersky, I have witnessed him be the “bigger person” and not retaliate.</p>
<p>This professional behavior stands in stark contrast to the behavior of Travis Brown.</p>
<p>Travis Brown first personally insulted Odersky when Odersky indicated that the Scala programming language website would freely link to the event <a href="https://github.com/scala/scala-lang/pull/1088#issuecomment-549023621">Functional Scala 2019</a>, without regard for Travis Brown’s personal vendetta.</p>
<p>Yet, when <a href="https://github.com/tpolecat/doobie/pull/1587#issuecomment-961415475">Odersky commented</a> on TypeLevel’s recent behavior, with a sentiment <a href="https://www.reddit.com/r/scala/comments/qu8wag/typelevel_on_recent_events/">shared by many in the Scala community</a>, Travis Brown went on a rampage and started viciously attacking Odersky.</p>
<p>The personal attacks did not stop with Odersky.</p>
<h2 id="scala-oss-developers">Scala OSS Developers</h2>
<p>Travis Brown expanded his targets to include a Muslim woman of color from Africa and an Orthodox Jew, both of whom contribute to ZIO, and both of whom happen to be my personal friends.</p>
<p>In a familiar pattern of online abuse, Travis Brown recovered deleted tweets from these people, archived them permanently without their consent, and started posting this material publicly, with grotesque and twisted interpretations.</p>
<p>In a separate incident, Travis Brown even went so far as to try to correlate a developer’s pseudonymous identity across different social media platforms and dox them.</p>
<p>All of this behavior is in clear violation of the Scala Code of Conduct, which most events and open source projects in the Scala community have adopted. One developer reported these violations to the TypeLevel moderators, who then deleted the report at the request of Travis Brown.</p>
<p>Travis Brown’s abusive behavior resulted in extreme psychological distress for these individuals. The woman I mentioned was afraid for her own physical safety as she walked the street of Berlin, where Travis Brown lives.</p>
<p>During the worst of these attacks, I feared someone might end up dead because of Travis Brown.</p>
<h2 id="taking-a-stand">Taking a Stand</h2>
<p>I am not responsible for the destructive behavior of Travis Brown. He alone is culpable for both his actions, and the consequences of his actions.</p>
<p>Nonetheless, in the past few days, I have wondered if my reluctance to speak up has contributed to the current environment, in which Travis Brown feels entitled to engage in this reprehensible behavior without fear of criticism or consequences.</p>
<p>Now, after being pushed too far, I believe it is time for me to take a stand.</p>
<p>Let me be as clear as I can possibly be:</p>
<ul>
<li><strong>Travis Brown’s online abuse, including his history of cyberstalking, online harassment, doxing, endorsement of violence, verbal abuse, and defamation, is completely unacceptable, and I condemn this behavior in the strongest possible terms.</strong></li>
<li><strong>Travis Brown’s attacks on Martin Odersky and other Scala open source developers are also unacceptable and without merit, and I condemn them unreservedly.</strong></li>
<li><strong>Encouraging, supporting, or enabling Travis Brown’s abusive behavior is also completely unacceptable, and I condemn it without hestitation.</strong></li>
<li><strong>Being abusive to Travis Brown, or engaging in the same behavior that he engages in, is also completely unacceptable, and I condemn it just as strongly.</strong></li>
</ul>
<p>I honestly hope that I never have to wade into this muck again.</p>
<p>I don’t enjoy it. I want to go back to creating beautiful and amazing technology that changes the world for the better, and empowering others to do the same.</p>
<p>But make no mistake about it: if anyone goes beyond just attacking me, to attacking the broader community, then I <em>will</em> take a stand, and I will <em>not</em> back down.</p>
<p><a href="https://degoes.net/articles/travis-brown-abuser">Supporting Martin Odersky & Other Scala OSS Developers</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on November 19, 2021.</p>https://degoes.net/articles/big-tent2021-11-11T00:00:00-07:002021-11-11T00:00:00-07:00John A De Goeshttps://degoes.netjohn@degoes.net<blockquote>
<p>“We but mirror the world. All the tendencies present in the outer world are to be found in the world of our body. If we could change ourselves, the tendencies in the world would also change. As a man changes his own nature, so does the attitude of the world change towards him. This is the divine mystery supreme. A wonderful thing it is and the source of our happiness. We need not wait to see what others do.” — Mahatma Gandhi</p>
</blockquote>
<p>Over the past week, I’ve seen a lot of pain and hurt in the Scala community, which has spilled over onto Reddit, Discord, Discourse, and Twitter, nearly all of it because developers are feeling targeted and attacked.</p>
<p>Inspired by the saying, <em>be the change you want to see in the world</em>, as well as excellent feedback from friends, I have decided to make a personal pledge, which I am calling the <em>Scala Pledge of Professionalism</em>, because it fosters professionalism within public Scala forums.</p>
<p>I’m making this pledge publicly for two reasons: one, so everyone understands the standards I will hold myself to; and two, so that anyone who feels so inspired can make their own pledge (maybe the same as mine, or maybe different).</p>
<h2 id="the-scala-pledge-of-professionalism">The Scala Pledge of Professionalism</h2>
<p>In all public forums across the Scala community, including all public Scala-related forums on Discord, Reddit, Discourse, Gitter, and Github, I hereby pledge the following:</p>
<ul>
<li>I will not insult, berate, or label developers in ways they would find insulting, nor will I impugn their character or judge their motivations. To the extent I must discuss behavior, I will keep my descriptions factual, using neutral language.</li>
<li>I will not berate the technical work of other Scala developers, and if I discuss or compare the work of Scala developers, I will do so respectfully and in an objective fashion, being fully prepared to publicly defend or recant any statements I make.</li>
<li>I will not take my religious, political, or other likely contentious personal beliefs into technical Scala conversations, but instead, I will do my best to ensure that every technical conversation remains focused purely on technical matters.</li>
<li>To the extent I have management or moderator abilities in any Scala forum, I will not tolerate bigotry, ad hominem, harassment, bullying, or career sabotage, but will work empathically and graciously yet firmly and decisively to ensure professionalism.</li>
<li>I will not break this pledge just because others break it, nor will I have exceptions for any Scala developers, regardless of their past or current behavior. I will not discriminate between Scala developers based on whether they make this pledge.</li>
</ul>
<p>Signed,</p>
<p><em>—John A. De Goes</em></p>
<p>P.S. If you so desire, feel free to sign this pledge by replying to this blog post, or by copying and pasting the pledge into your own blog or public space.</p>
<p><a href="https://degoes.net/articles/big-tent">Improving the Scala Community Through Personal Action</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on November 11, 2021.</p>https://degoes.net/articles/zio-professionalism2021-11-07T00:00:00-06:002021-11-07T00:00:00-06:00John A De Goeshttps://degoes.netjohn@degoes.net<p>In the past few months, the TypeLevel organization has become increasingly hostile toward the ZIO community (whose software directly competes with TypeLevel):</p>
<ul>
<li>Rob Norris <a href="https://github.com/tpolecat/doobie/pull/1587">removing</a> an existing Quill integration from Doobie because Quill decided to move to the ZIO organization</li>
<li>Rob Norris <a href="https://github.com/tpolecat/skunk/issues/438">opting to not help a user</a> specifically because they were using Skunk with ZIO</li>
<li>Ross Baker and Christopher Davenport <a href="https://github.com/http4s/http4s/issues/4718">rejecting a ZIO integration</a> on grounds that ZIO is “inconsistent with our values”</li>
<li>Rob Norris and Michael Pilquist referring to ZIO as <code class="language-plaintext highlighter-rouge">Z**</code> or <code class="language-plaintext highlighter-rouge">*IO</code>, with Rob Norris <a href="https://twitter.com/tpolecat/status/1424368733976416257">leaving</a> the <a href="https://discord.gg/scala">Scala Discord</a> because ZIO was <a href="https://twitter.com/nafg613/status/1424542295244873730">mentioned</a></li>
<li>Oscar Boykin <a href="https://twitter.com/flaviusbraz/status/1456615716619055106">attacking</a> both an Orthodox Jew and a Latino who contribute to Quill and ZIO</li>
</ul>
<p>As an open source developer myself, I am keenly aware of the fact that OSS is a “nights and weekends” hobby for most of us. We freely volunteer our time, often sacrificing other pursuits in order to contribute to open source and support our end-users.</p>
<p>As a result of this reality, I wholeheartedly support the right of every OSS developer to contribute on their own terms, whatever those terms may be. Non-paying users of free software should <em>not</em> get to dictate these terms.</p>
<p>At the same time, I recognize that the actions of the TypeLevel organization may have undesirable side-effects on the Scala community:</p>
<ul>
<li>They undermine the trust that end-users and companies place in Scala OSS</li>
<li>They increase the risk involved in deploying solutions based on Scala OSS</li>
<li>They decrease the network value of Scala OSS</li>
<li>They make Scala look unprofessional to many developers, especially compared to Java or other ecosystems where major OSS projects would never behave in such a fashion</li>
</ul>
<p>As a passionate fan of the Scala programming language, a well-known advocate for functional programming, a leader in the ZIO open source ecosystem, and a founder of a <a href="https://ziverge.com">company devoted to Scala OSS</a>, I want Scala to prosper.</p>
<p>I have no ability to influence the TypeLevel organization. However, as <a href="https://en.wikipedia.org/wiki/Benevolent_dictator_for_life">BDFL</a> of the <a href="https://github.com/zio">ZIO organization</a>, I am in a position where I can learn from these behaviors, and implement changes within the ZIO organization that I believe will benefit the whole Scala community.</p>
<p>Effective immediately, I hereby codify and will support the following:</p>
<ul>
<li><strong>Pro-Community</strong>. All ZIO projects hosted within the <a href="https://github.com/zio">official ZIO organization</a> will gladly accept and host integrations for Akka, TypeLevel, and other Scala or JVM ecosystems, without consideration of the relationships, dispositions, or politics between these projects and those ecosystems, and they will provide non-discriminatory support for end-users, regardless of their disposition to or affiliation or association with other Scala community members or ecosystems.</li>
<li><strong>Pro-Professionalism</strong>. Although behavior within ZIO organization projects is already governed by the <a href="https://scala-lang.org/conduct/">Scala Code of Conduct</a>, I want to strengthen this code of conduct by making it clear that ad hominem and career sabotage has no place in the ZIO community. Projects in the ZIO organization exist only to help developers solve problems, regardless of their race, sexual orientation, gender identity, or disposition to or affiliation or association with other Scala community members or ecosystems.</li>
</ul>
<p>ZIO is already doing a great job at being <em>pro-community</em>, as ZIO JSON hosts an integration for the TypeLevel projects <em>http4s</em> and <em>Refined</em>, and ZIO Core hosts integrations for <em>Cats Effect 2</em>, <em>Cats Effect 3</em>, and <em>Akka Cluster</em> (among other integrations too numerous to list). However, I believe that making this behavior official will further increase trust and expand integrations, and also clearly set expectations for new projects coming into the organization.</p>
<p>As for being <em>pro-professionalism</em>, within ZIO official spaces (Github, Discord, etc.), I have only ever seen welcoming, inclusive, and non-discriminatory behavior, without ad hominem or career sabotage. But explicitly committing to this high-standard of professionalism can only help to set expectations and provide guidance for leaders as the organization continues to grow.</p>
<p>I encourage other Scala open source organizations to follow ZIO’s lead, and adopt these <em>pro-community</em> and <em>pro-professsionalism</em> measures, without imposing any negative repercussions on Scala open source contributors (including TypeLevel) who choose to take a different path (OSS entitlement is a <a href="https://medium.com/@fommil/the-open-source-entitlement-complex-bcb718e2326d">real problem</a>).</p>
<p>As <a href="https://en.wikipedia.org/wiki/Martin_Odersky">Martin Odersky</a> once wrote, <a href="https://www.scala-lang.org/blog/2019/05/02/community.html">Scala is a big tent</a>. It is not necessary that open source contributors have the same views or even like each other. But if we put Scala first, by being pro-community and pro-professionalism, we can find a way to co-exist peacefully inside this big tent, and together, we can, in different ways and with different audiences, show the world that Scala is a force to reckon with.</p>
<p><a href="https://degoes.net/articles/zio-professionalism">Improvements to the ZIO Organization</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on November 07, 2021.</p>https://degoes.net/articles/functional-design2020-08-18T00:00:00-06:002020-08-18T00:00:00-06:00John A De Goeshttps://degoes.netjohn@degoes.net<p>Functional programming doesn’t need to be complex, confusing, or theoretical.</p>
<p>Instead, functional programming can be simple, natural, and practical, helping you solve problems
with more power and more joy than ever before.</p>
<p>A key to unlocking this potential is understanding <em>functional design</em>, a framework for applying
functional programming to real world problems that I have been teaching at
<a href="https://ziverge.com">Ziverge</a> as well as on my <a href="https://patreon.com">Patreon mentorship program</a>.</p>
<p>Functional design helps you put the “practical” in functional programming, distilling the essence of
functional programming into a toolkit that you can use anywhere, any time, in any programming
language, and with either greenfield or legacy code bases.</p>
<p>Although developing skills in functional design requires work, time, and practice, the concepts are
few and simple—perfect content for a blog post, in fact.</p>
<p>So in this post, I’m going to introduce you to functional design!</p>
<h2 id="elements-of-functional-design">Elements of Functional Design</h2>
<p>In <em>functional design</em>, we create <em>immutable data types</em> that <em>model</em> solutions to problems in some
<em>domain of interest</em>, and equip these data types with <em>constructors</em>, which let us build <em>simple
solutions</em>, and <em>composable operators</em>, which let us transform and combine solutions in a
predictable way, so that we can use a small set of primitive operators and constructors to solve all
possible problems in the domain of interest.</p>
<p>That’s quite a mouthful! So let’s break it down, one bite at a time:</p>
<ul>
<li><em>Functional design</em>. Functional design is an alternative to object-oriented design. Both try to
solve business problems in ways that reduce defects and lower the cost of maintenance.</li>
<li><em>Immutable data types</em>. Functional programming uses immutable data types, which are a better fit
for concurrent applications and which invert control (the functions that you call cannot modify the
data that you pass them), making code maintenance easier and less risky.</li>
<li><em>Model</em>. Most functional code doesn’t solve problems <em>directly</em>. Instead, it creates
<em>models</em> of solutions to problems, which are later executed (or <em>interpreted</em>). For example, a
route in a REST web server models handling a URL path, which is later executed on each request.
Models are always represented using immutable data types.</li>
<li><em>Solutions</em>. At every level of an application, there are problems that require solutions. Data
needs to be validated, requests need responses, business logic needs to trigger emails, numbers
need to be aggregated, and so forth.</li>
<li><em>Domain of interest</em>. In most applications, there are dozens to hundreds of different domains of
interest, each of which contains related entities and similar problems that we would like to solve
in a way that is easy to test and maintain.</li>
<li><em>Constructors</em>. Constructors let us build values using our <em>immutable data types</em> that model
solutions to <em>simple</em> problems.</li>
<li><em>Composable operators</em>. Operators let us transform and combine solutions to subproblems so we can
make values that model solutions to larger, much more complex problems.</li>
</ul>
<p>Functional design works great with legacy code and greenfield code, although you may use slightly
different techniques in each of these cases.</p>
<p>A <em>functional domain</em> is a shorthand way to refer to some particular domain of interest, together
with its immutable data type that models solutions to problems, and the model’s constructors and
operators.</p>
<p>In the next section, we’ll take a look at a few example functional domains.</p>
<h2 id="example-domains">Example Domains</h2>
<p>Functional design is at the heart of all great functional programming libraries.</p>
<p>Below are a few of the examples you may have already seen:</p>
<ul>
<li><strong>Parser Combinators</strong>. Parser combinator libraries are designed for the domain of parsing text
formats into data structures. A parser itself is a model that describes how to parse some piece of
data from a character input sequence. Parser constructors let you build simple parsers, while
operators let you transform and combine parsers so you can parse more complicated formats. You can
execute parsers, which involves feeding them input to produce either an error or a value that
captures the structure in the parsed data.</li>
<li><strong>Functional Effects</strong>. Functional effect libraries like <a href="https://zio.dev">ZIO</a> are designed for
the domain of concurrent programming, with safe error management and resource handling. An effect is
a model that describes how to execute a series of sequential and concurrent operations. Effect
constructors let you build simple (non-concurrent) operations, while operators let you transform and
combine effects so you can model much more sophisticated concurent tasks. You can execute effects,
which performs all the operations they describe.</li>
<li><strong>Optics</strong>. Optics libraries like <a href="https://www.optics.dev/Monocle/">Monocle</a> are designed for the
domain of accessing and transforming immutable data structures. An optic is a model that describes
how to zoom into a piece of data, and retrieve or modify it in some fashion. Optic constructors let
you build simple accessors and modifiers, while operators let you transform and combine optics so
you can model access and modification on gigantic, highlly nested data structures. You can
“execute” optics, which involves feeding them data and transformations to get transformed data.</li>
<li><strong>Streams</strong>. Stream libraries like <a href="https://zio.dev">ZIO Stream</a> are designed for the domain of
concurrent streaming data. A stream is a model of a concurrently generated sequence of values.
Stream constructors let you build simple streams, while operators let you transform and combine
streams so you can model sophisticated data pipelines. You can execute streams, which involves
reading elements from their source locations, and concurrently transforming and combining them as
per the model.</li>
</ul>
<p>I will take one example functional domain in ZIO and drill down to the level of actual code, to make
everything as concrete as possible.</p>
<h3 id="in-depth-zio-schedule">In Depth: ZIO Schedule</h3>
<p><em>ZIO Schedule</em> is an immutable data type that models a solution to the problem of specifying a
finite or infinite recurring schedule.</p>
<p>As with every functional domain, there is a model, constructors, and operators.</p>
<p>The model is the data type <code class="language-plaintext highlighter-rouge">zio.Schedule[Env, In, Out]</code>, which describes a schedule that runs in
some environment <code class="language-plaintext highlighter-rouge">Env</code>, requires inputs of type <code class="language-plaintext highlighter-rouge">In</code>, and produces outputs of type <code class="language-plaintext highlighter-rouge">Out</code>.</p>
<p>When schedules are used to retry failed effects, the input type to the schedule is the error type of
the effect. When schedules are used to repeat successful effects, the input type to the schedule is
the success type of the effect.</p>
<p>Like all models, a <code class="language-plaintext highlighter-rouge">Schedule</code> doesn’t actually do anything except describe a recurring schedule
(models <em>describe</em>, they don’t <em>do</em>). To execute a schedule model, you can call the <code class="language-plaintext highlighter-rouge">retry</code> or
<code class="language-plaintext highlighter-rouge">repeat</code> methods on any ZIO effect, and feed them the schedule.</p>
<p>The constructors give us schedules that model solutions to very simple problems. For example, if we
wish to describe a recurrence on Thursday, we can use the <code class="language-plaintext highlighter-rouge">Schedule.dayOfWeek</code> constructor:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">onThursday</span> <span class="k">=</span> <span class="nv">Schedule</span><span class="o">.</span><span class="py">dayOfWeek</span><span class="o">(</span><span class="nv">DayOfWeek</span><span class="o">.</span><span class="py">THURSDAY</span><span class="o">)</span></code></pre></figure>
<p>Similarly, if we wish to describe a recurrence at 6:00 AM and 12:00 PM, then we can use the
<code class="language-plaintext highlighter-rouge">Schedule.hourOfDay</code> constructor:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">sixAndTwelve</span> <span class="k">=</span> <span class="nv">Schedule</span><span class="o">.</span><span class="py">hourOfDay</span><span class="o">(</span><span class="mi">6</span><span class="o">,</span> <span class="mi">12</span><span class="o">)</span></code></pre></figure>
<p>Like all models in functional design, the schedule model has operators that let us transform and
combine schedules, so we can solve bigger problems from solutions to much simpler subproblems.</p>
<p>For example, if we want a schedule that recurs on Thursdays at 6:00 AM and 12:00 PM, then we can
use the intersection operator, which produces the intersection of two schedules:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">thursdaysSixAndTwelve</span> <span class="k">=</span>
<span class="n">onThursday</span> <span class="o">&&</span> <span class="n">sixAndTwelve</span></code></pre></figure>
<p>Similarly, if we want to modify this schedule, so that it produces a constant output (say, the unit
value <code class="language-plaintext highlighter-rouge">()</code>), we can use the <code class="language-plaintext highlighter-rouge">map</code> method, which allows us to produce a new schedule with a
transformed output:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">finalSchedule</span> <span class="k">=</span> <span class="nv">thursdaysSixAndTwelve</span><span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="k">_</span> <span class="k">=></span> <span class="o">())</span></code></pre></figure>
<p>Because <em>ZIO Schedule</em> is a functional domain, it has the ability to solve a huge number of
scheduling problems using a small set of constructors and operators. Thanks to following the
principles of functional design, schedules are fully testable, easy to understand, and easy to
modify in the face of changing business requirements.</p>
<p>Now that you’ve seen an example of functional design up close, we’re going to build our own simple
functional domain and explore different choices for encoding the model.</p>
<h2 id="email-filtering">Email Filtering</h2>
<p>Let’s say we’re building an email web application. Users of the application need the ability to
construct email filters, so they can trigger actions like forwarding emails, moving emails to
certain folders, and deleting emails.</p>
<p>There are a huge number of possible filters that users might want to construct. Moreover, as
developers of the application, we want the ability to support these numerous use cases in a way that
is highly testable and has a low cost of maintenance.</p>
<p>To achieve these benefits, we will use functional design techniques.</p>
<h3 id="the-model">The Model</h3>
<p>The first thing we need in this new functional domain is a <em>model</em>. This model will be an immutable
data type (like <em>all</em> models), and will describe a particular filter that the user has constructed
and wishes to apply to incoming emails.</p>
<p>One of the ways we want to <em>execute</em> this model is to apply it to an email and see if the filter
matches the email. This consideration can influence the design of our model.</p>
<p>In functional design, there are two ways to encode any model:</p>
<ul>
<li><strong>Executable Encoding</strong>. In this encoding, we express every constructor and operator for our
model in terms of its execution.</li>
<li><strong>Declarative Encoding</strong>. In this encoding, we express every constructor and operator for our
model as pure data in a recursive tree structure.</li>
</ul>
<p>The first encoding is sometimes called <em>final</em> (because the constructors and operators are expressed
in terms of its final executed form, which is predetermined in advance), while the latter is
sometimes called <em>initial</em> (because given this initial form, the model can later be executed into
any form, even those not predetermined in advance).</p>
<p>Let’s explore both encodings of an email filter.</p>
<h3 id="executable-encoding">Executable Encoding</h3>
<p>Executable encodings are represented using <code class="language-plaintext highlighter-rouge">case class</code> types or open <code class="language-plaintext highlighter-rouge">trait</code> types that store
a bunch of functions, each of which executes the model in some predetermined way.</p>
<p>In our case, we want to execute an email filter by applying it to an email, and seeing if the filter
matches the email. So to create an executable encoding of an <code class="language-plaintext highlighter-rouge">EmailFilter</code>, we will define a
<code class="language-plaintext highlighter-rouge">case class</code>, and store a function that takes an email and returns a boolean value:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">EmailFilter</span><span class="o">(</span><span class="n">matches</span><span class="k">:</span> <span class="kt">Email</span> <span class="o">=></span> <span class="nc">Boolean</span><span class="o">)</span></code></pre></figure>
<p>To execute this model, we need only call the <code class="language-plaintext highlighter-rouge">matches</code> function that we have stored in the model,
and provide it an email, which will then give us a boolean value that tells us whether or not the
email filter matches the specified email.</p>
<p>We may now proceed to implement constructors, which build solutions to simple problems, such as
matching emails that have some subject:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">subjectContains</span><span class="o">(</span><span class="n">phrase</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">EmailFilter</span> <span class="o">=</span>
<span class="nc">EmailFilter</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">subject</span><span class="o">.</span><span class="py">contains</span><span class="o">(</span><span class="n">phrase</span><span class="o">))</span></code></pre></figure>
<p>In a realistic example, other constructors would be needed, which would allow filtering on bodies,
or filtering on the recipient list, or filtering on the send date (for example).</p>
<p>Constructors only solve simple problems. To solve more complex problems, we need operators that
allow transforming and composing models.</p>
<p>We can add these methods directly inside the <code class="language-plaintext highlighter-rouge">EmailFilter</code> class:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">EmailFilter</span><span class="o">(</span><span class="n">matches</span><span class="k">:</span> <span class="kt">Email</span> <span class="o">=></span> <span class="nc">Boolean</span><span class="o">)</span> <span class="o">{</span> <span class="n">self</span> <span class="k">=></span>
<span class="k">def</span> <span class="nf">&&</span><span class="o">(</span><span class="n">that</span><span class="k">:</span> <span class="kt">EmailFilter</span><span class="o">)</span><span class="k">:</span> <span class="kt">EmailFilter</span> <span class="o">=</span>
<span class="nc">EmailFilter</span><span class="o">(</span><span class="n">email</span> <span class="k">=></span> <span class="nv">self</span><span class="o">.</span><span class="py">matches</span><span class="o">(</span><span class="n">email</span><span class="o">)</span> <span class="o">&&</span> <span class="nv">that</span><span class="o">.</span><span class="py">matches</span><span class="o">(</span><span class="n">email</span><span class="o">))</span>
<span class="k">def</span> <span class="nf">||</span><span class="o">(</span><span class="n">that</span><span class="k">:</span> <span class="kt">EmailFilter</span><span class="o">)</span><span class="k">:</span> <span class="kt">EmailFilter</span> <span class="o">=</span>
<span class="nc">EmailFilter</span><span class="o">(</span><span class="n">email</span> <span class="k">=></span> <span class="nv">self</span><span class="o">.</span><span class="py">matches</span><span class="o">(</span><span class="n">email</span><span class="o">)</span> <span class="o">||</span> <span class="nv">that</span><span class="o">.</span><span class="py">matches</span><span class="o">(</span><span class="n">email</span><span class="o">))</span>
<span class="k">def</span> <span class="nf">unary_!</span> <span class="k">:</span> <span class="kt">EmailFilter</span> <span class="o">=</span>
<span class="nc">EmailFilter</span><span class="o">(</span><span class="n">email</span> <span class="k">=></span> <span class="o">!</span><span class="nv">self</span><span class="o">.</span><span class="py">matches</span><span class="o">(</span><span class="n">email</span><span class="o">))</span>
<span class="o">}</span></code></pre></figure>
<p>These two binary operators and one unary operator are enough to solve quite a few interesting
problems in email filtering. For example, the following filter accepts emails whose subject contains
the words “discount” or “clearance”, and which do not contain the word “liquidation”:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">filter</span> <span class="k">=</span>
<span class="o">(</span><span class="nf">subjectContains</span><span class="o">(</span><span class="s">"discount"</span><span class="o">)</span> <span class="o">||</span> <span class="nf">subjectContains</span><span class="o">(</span><span class="s">"clearance"</span><span class="o">))</span> <span class="o">&&</span>
<span class="o">!</span><span class="nf">subjectContains</span><span class="o">(</span><span class="s">"liquidation"</span><span class="o">)</span></code></pre></figure>
<p>The executable encoding is nice and straightforward: it’s obvious what an email filter is, even to
someone who is not very familiar with functional design, because every email filter is expressed
directly in terms of its execution.</p>
<h3 id="declarative-encoding">Declarative Encoding</h3>
<p>Declarative encodings are represented using <code class="language-plaintext highlighter-rouge">sealed trait</code> types or <code class="language-plaintext highlighter-rouge">enum</code> types (Scala 3), which
have as many subtypes (or <em>cases</em>) as the model has primitive constructors and operators. The
purpose of these subtypes is to capture the arguments of every constructor and operator, and store
them as pure data in a recursive data type.</p>
<p>In our case, if we want to replicate the functionality of the previous executable encoding, then we
need four subtypes of a <code class="language-plaintext highlighter-rouge">sealed trait</code>, one for the subject constructor, one for the unary operator,
and two more for the binary operators:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">EmailFilter</span> <span class="o">{</span> <span class="n">self</span> <span class="k">=></span>
<span class="k">def</span> <span class="nf">&&</span><span class="o">(</span><span class="n">that</span><span class="k">:</span> <span class="kt">EmailFilter</span><span class="o">)</span><span class="k">:</span> <span class="kt">EmailFilter</span> <span class="o">=</span> <span class="nc">And</span><span class="o">(</span><span class="n">self</span><span class="o">,</span> <span class="n">that</span><span class="o">)</span>
<span class="k">def</span> <span class="nf">||</span><span class="o">(</span><span class="n">that</span><span class="k">:</span> <span class="kt">EmailFilter</span><span class="o">)</span><span class="k">:</span> <span class="kt">EmailFilter</span> <span class="o">=</span> <span class="nc">Or</span><span class="o">(</span><span class="n">self</span><span class="o">,</span> <span class="n">that</span><span class="o">)</span>
<span class="k">def</span> <span class="nf">unary_!</span> <span class="k">:</span> <span class="kt">EmailFilter</span> <span class="o">=</span> <span class="nc">Not</span><span class="o">(</span><span class="n">self</span><span class="o">)</span>
<span class="o">}</span>
<span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">SubjectContains</span><span class="o">(</span><span class="n">phrase</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">EmailFilter</span>
<span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">And</span><span class="o">(</span><span class="n">left</span><span class="k">:</span> <span class="kt">EmailFilter</span><span class="o">,</span> <span class="n">right</span><span class="k">:</span> <span class="kt">EmailFilter</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">EmailFilter</span>
<span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Or</span><span class="o">(</span><span class="n">left</span><span class="k">:</span> <span class="kt">EmailFilter</span><span class="o">,</span> <span class="n">right</span><span class="k">:</span> <span class="kt">EmailFilter</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">EmailFilter</span>
<span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Not</span><span class="o">(</span><span class="n">value</span><span class="k">:</span> <span class="kt">EmailFilter</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">EmailFilter</span>
<span class="k">def</span> <span class="nf">subjectContains</span><span class="o">(</span><span class="n">phrase</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">EmailFilter</span> <span class="o">=</span>
<span class="nc">SubjectContains</span><span class="o">(</span><span class="n">phrase</span><span class="o">)</span></code></pre></figure>
<p>As with all declarative encodings, the subtypes just store the arguments to the constructors and the
operators—they don’t actually do anything except “record” the way email filters are
constructed, transformed, and composed.</p>
<p>If we take the previous example <code class="language-plaintext highlighter-rouge">filter</code>, then we can see it ends up building a tree:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="nc">And</span><span class="o">(</span>
<span class="nc">Or</span><span class="o">(</span><span class="nc">SubjectContains</span><span class="o">(</span><span class="s">"discount"</span><span class="o">),</span> <span class="nc">SubjectContains</span><span class="o">(</span><span class="s">"clearance"</span><span class="o">)),</span>
<span class="nc">Not</span><span class="o">(</span><span class="nc">SubjectContains</span><span class="o">(</span><span class="s">"liquidation"</span><span class="o">))</span>
<span class="o">)</span></code></pre></figure>
<p>This tree can be inspected and transformed, like any data tree.</p>
<p>Now because a model using the declarative encoding is pure data, there is no built-in way to execute
the model: we have no way to test an email to see if it matches a filter. However, it’s easy to
write a standalone function that can execute the model (called an <em>interpreter</em> or <em>executor</em> of the
model).</p>
<p>The following executor takes a model, and tests an email to see if it matches the filter:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">matches</span><span class="o">(</span><span class="n">filter</span><span class="k">:</span> <span class="kt">EmailFilter.</span> <span class="kt">email:</span> <span class="kt">Email</span><span class="o">)</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span>
<span class="n">filter</span> <span class="k">match</span> <span class="o">{</span>
<span class="k">case</span> <span class="nc">And</span><span class="o">(</span><span class="n">l</span><span class="o">,</span> <span class="n">r</span><span class="o">)</span> <span class="k">=></span> <span class="nf">matches</span><span class="o">(</span><span class="n">l</span><span class="o">,</span> <span class="n">email</span><span class="o">)</span> <span class="o">&&</span> <span class="nf">matches</span><span class="o">(</span><span class="n">r</span><span class="o">,</span> <span class="n">email</span><span class="o">)</span>
<span class="k">case</span> <span class="nc">Or</span><span class="o">(</span><span class="n">l</span><span class="o">,</span> <span class="n">r</span><span class="o">)</span> <span class="k">=></span> <span class="nf">matches</span><span class="o">(</span><span class="n">l</span><span class="o">,</span> <span class="n">email</span><span class="o">)</span> <span class="o">||</span> <span class="nf">matches</span><span class="o">(</span><span class="n">r</span><span class="o">,</span> <span class="n">email</span><span class="o">)</span>
<span class="k">case</span> <span class="nc">Not</span><span class="o">(</span><span class="n">v</span><span class="o">)</span> <span class="k">=></span> <span class="o">!</span><span class="nf">matches</span><span class="o">(</span><span class="n">v</span><span class="o">,</span> <span class="n">email</span><span class="o">)</span>
<span class="k">case</span> <span class="nc">SubjectContains</span><span class="o">(</span><span class="n">phrase</span><span class="o">)</span> <span class="k">=></span> <span class="nv">email</span><span class="o">.</span><span class="py">subject</span><span class="o">.</span><span class="py">contains</span><span class="o">(</span><span class="n">phrase</span><span class="o">)</span>
<span class="o">}</span></code></pre></figure>
<p>The declarative encoding has a layer of indirection, which adds additional ceremony. However, it is
a nice way to think about models in general, because it is obvious they describe solutions to
problems—the execution of the model requires a separate traversal of the data structure.</p>
<h2 id="new-interpreters">New Interpreters</h2>
<p>In the previous examples, we executed the <code class="language-plaintext highlighter-rouge">EmailFilter</code> model by passing an email, and we got back
a boolean value indicating whether or not the filter matched the email.</p>
<p>However, that’s just one of many ways we might want to execute a model. For example, we might also
want to execute the model into a string, which tells us how the email filter was constructed,
transformed, and composed (for debug purposes or rendering to users).</p>
<p>We can do this for both the executable and the declarative encodings.</p>
<p>For the executable encoding, we just add another function of type <code class="language-plaintext highlighter-rouge">() => String</code> to the
<code class="language-plaintext highlighter-rouge">case class</code>. This second executor creates a string representation of the email filter:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">EmailFilter</span><span class="o">(</span><span class="n">matches</span><span class="k">:</span> <span class="kt">Email</span> <span class="o">=></span> <span class="nc">Boolean</span><span class="o">,</span> <span class="n">describe</span><span class="k">:</span> <span class="o">()</span> <span class="o">=></span> <span class="nc">String</span><span class="o">)</span> <span class="o">{</span> <span class="n">self</span> <span class="k">=></span>
<span class="k">def</span> <span class="nf">&&</span><span class="o">(</span><span class="n">that</span><span class="k">:</span> <span class="kt">EmailFilter</span><span class="o">)</span><span class="k">:</span> <span class="kt">EmailFilter</span> <span class="o">=</span>
<span class="nc">EmailFilter</span><span class="o">(</span>
<span class="n">email</span> <span class="k">=></span> <span class="nv">self</span><span class="o">.</span><span class="py">matches</span><span class="o">(</span><span class="n">email</span><span class="o">)</span> <span class="o">&&</span> <span class="nv">that</span><span class="o">.</span><span class="py">matches</span><span class="o">(</span><span class="n">email</span><span class="o">),</span>
<span class="o">()</span> <span class="k">=></span> <span class="n">s</span><span class="s">"(${self.describe()} && ${that.describe})"</span><span class="o">)</span>
<span class="k">def</span> <span class="nf">||</span><span class="o">(</span><span class="n">that</span><span class="k">:</span> <span class="kt">EmailFilter</span><span class="o">)</span><span class="k">:</span> <span class="kt">EmailFilter</span> <span class="o">=</span>
<span class="nc">EmailFilter</span><span class="o">(</span>
<span class="n">email</span> <span class="k">=></span> <span class="nv">self</span><span class="o">.</span><span class="py">matches</span><span class="o">(</span><span class="n">email</span><span class="o">)</span> <span class="o">||</span> <span class="nv">that</span><span class="o">.</span><span class="py">matches</span><span class="o">(</span><span class="n">email</span><span class="o">),</span>
<span class="o">()</span> <span class="k">=></span> <span class="n">s</span><span class="s">"(${self.describe()} || ${that.describe})"</span><span class="o">))</span>
<span class="k">def</span> <span class="nf">unary_!</span> <span class="k">:</span> <span class="kt">EmailFilter</span> <span class="o">=</span>
<span class="nc">EmailFilter</span><span class="o">(</span><span class="n">email</span> <span class="k">=></span> <span class="o">!</span><span class="nv">self</span><span class="o">.</span><span class="py">matches</span><span class="o">(</span><span class="n">email</span><span class="o">),</span>
<span class="o">()</span> <span class="k">=></span> <span class="n">s</span><span class="s">"!${self.describe()}"</span><span class="o">))</span>
<span class="o">}</span>
<span class="k">def</span> <span class="nf">subjectContains</span><span class="o">(</span><span class="n">phrase</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">EmailFilter</span> <span class="o">=</span>
<span class="nc">EmailFilter</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">subject</span><span class="o">.</span><span class="py">contains</span><span class="o">(</span><span class="n">phrase</span><span class="o">),</span> <span class="o">()</span> <span class="k">=></span> <span class="n">s</span><span class="s">"(subject contains ${phrase})"</span><span class="o">)</span></code></pre></figure>
<p>In general, if we want <code class="language-plaintext highlighter-rouge">n</code> distinct ways to execute a model, then our executable encoding will have
<code class="language-plaintext highlighter-rouge">n</code> functions all stored inside the <code class="language-plaintext highlighter-rouge">case class</code> (or <code class="language-plaintext highlighter-rouge">n</code> methods inside an open <code class="language-plaintext highlighter-rouge">trait</code>).</p>
<p>Every time we add a new interpreter to a model expressed with the executable encoding, we have to
update all the code that uses the constructor for the model, which could potentially be a lot of
code. However, the maintenance burden can be minimized by creating <em>derived</em> operators and
constructors, which are expressed in terms of other constructors and operators (and not the primary
constructor for our <code class="language-plaintext highlighter-rouge">case class</code>).</p>
<p>For the declarative encoding, because the model is just data, if we want another interpreter, we can
just add another standalone function:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">describe</span><span class="o">(</span><span class="n">filter</span><span class="k">:</span> <span class="kt">EmailFilter</span><span class="o">)</span><span class="k">:</span> <span class="kt">String</span> <span class="o">=</span>
<span class="n">filter</span> <span class="k">match</span> <span class="o">{</span>
<span class="k">case</span> <span class="nc">And</span><span class="o">(</span><span class="n">l</span><span class="o">,</span> <span class="n">r</span><span class="o">)</span> <span class="k">=></span> <span class="n">s</span><span class="s">"(${describe(l)} && ${describe(r)})"</span>
<span class="k">case</span> <span class="nc">Or</span><span class="o">(</span><span class="n">l</span><span class="o">,</span> <span class="n">r</span><span class="o">)</span> <span class="k">=></span> <span class="n">s</span><span class="s">"(${describe(l)} || ${describe(r)})"</span>
<span class="k">case</span> <span class="nc">Not</span><span class="o">(</span><span class="n">v</span><span class="o">)</span> <span class="k">=></span> <span class="n">s</span><span class="s">"!${describe(v)}"</span>
<span class="k">case</span> <span class="nc">SubjectContains</span><span class="o">(</span><span class="n">phrase</span><span class="o">)</span> <span class="k">=></span> <span class="n">s</span><span class="s">"(subject contains ${phrase})"</span>
<span class="o">}</span></code></pre></figure>
<p>The ease with which one can add new ways to execute a model is indeed a strength of the
declarative encoding.</p>
<h3 id="encoding-tradeoffs">Encoding Tradeoffs</h3>
<p>Executable and declarative encodings have <em>opposite</em> strengths and weaknesses.</p>
<p>In the executable encoding, new constructors and operators can be added freely, without updating
any existing code. However, new interpreters cannot be added without updating all existing
constructors and operators.</p>
<p>In the declarative encoding, new interpreters can be added freely, without updating any existing
code. However, new (primitive) operators and constructors cannot be added without updating all
existing interpreters.</p>
<p>In functional programming, the executable and declarative encoding are considered “duals”: they are
mirror images of each other, sitting at opposite ends of the same spectrum.</p>
<p>In both executable and declarative encodings, you can add new operators and constructors; and in
both executable and declarative encodings, you can add new interpreters, which execute the model
in some way (for example, testing to see if an email filter matches an email).</p>
<p>Now, as pure data, the declarative encoding is amenable to optimizations that are not possible with
the executable encoding (at least, not without essentially reinventing data). So for cases where
performance is critical, the declarative encoding can sometimes have an edge.</p>
<p>In addition, the declarative encoding is ideal when persistence of the model is required: as pure
data, it is straightforward to define a way to save and restore a model value, using a relational
database or other persistence scheme.</p>
<p>Meanwhile, the executable encoding tends to be a better fit for legacy code, because it is very
lean and can seamlessly reuse existing (impure) interfaces and classes. For example, if we have a
lot of code using an <code class="language-plaintext highlighter-rouge">InputStream</code>, we can write our own simple functional stream like so:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Stream</span><span class="o">(</span><span class="n">create</span><span class="k">:</span> <span class="o">()</span> <span class="o">=></span> <span class="nc">InputStream</span><span class="o">)</span></code></pre></figure>
<p>Such a data type can have many constructors and operators that let us program in a more declarative
fashion (without worrying about resource handling or error propagation for intermediate input
streams), while still maintaining easy interop with existing code bases.</p>
<h2 id="where-from-here">Where From Here</h2>
<p>Functional design is an immensely powerful tool for making functional programming practical. It lets
you solve real world problems in a way that makes your code easy to test, cheap to maintain, and fun
to work on.</p>
<p>Although this post introduces the basics, there’s much more beyond what I’ve talked about here,
including how to make functional domains more type safe (using generics, phantom types, and type-
level programming); how to design good constructors and operators using the principles of
orthogonality, expressivity, and composability; common patterns of transformation and
composition; and identifying functional domains inside business applications.</p>
<p>If you’ve enjoyed this post and would like to learn more, check out my
<a href="https://github.com/jdegoes/functional-design">exercises on Github</a>, sign up for my next workshop
at <a href="https://ziverge.com">Ziverge</a>, or join my Spartan tier on <a href="https://patreon.com/jdegoes">Patreon</a>.</p>
<p>Those for whom this post was an easy read are encouraged to explore
<a href="https://en.wikipedia.org/wiki/Expression_problem">the expression problem</a>,
<a href="https://www.cs.utexas.edu/~wcook/Drafts/2012/ecoop2012.pdf">object algebras</a>,
and <a href="http://okmij.org/ftp/tagless-final/index.html">tagless-final style</a>, all of which provide more
background on the tradeoffs of different encodings.</p>
<p>That’s all for now!</p>
<p><a href="https://degoes.net/articles/functional-design">An Introduction to Functional Design</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on August 18, 2020.</p>https://degoes.net/articles/zio-1.02020-08-03T00:00:00-06:002020-08-03T00:00:00-06:00John A De Goeshttps://degoes.netjohn@degoes.net<p>On June 5th, 2017, I <a href="https://github.com/scalaz/scalaz/issues/1399">opened an issue</a> in the Scalaz repository on Github. I argued Scalaz 8 needed a powerful and fast data type for async programming, and encouraged to contribute to Scalaz by my friend Vincent Marquez, I volunteered to build one.</p>
<p>This was not my first foray into async and concurrent programming. Previously, I had designed and built the first version of <a href="https://pursuit.purescript.org/packages/purescript-aff/5.1.2">Aff</a>, and assisted the talented Nathan Faubion with the second iteration. The library quickly became the number one solution for async and concurrent programming in the Purescript ecosystem.</p>
<p>Further back, I had written a Future data type for Scala, which I abandoned after the Scala standard library incorporated a much better version. Before that, I wrote a Promise data type for haXe, and a Raincheck data type for Java. In every case, I made mistakes, and subsequently learned to make new mistakes!</p>
<p>That sunny afternoon in June 2017, I imagined spending a few months developing this new data type, at which point I would wrap it up in a tidy bow, and hand it over to the Scalaz organization.</p>
<p>What happened next, I never could have imagined.</p>
<h2 id="the-birth-of-zio">The Birth of ZIO</h2>
<p>As I spent some free time working on the core of the new project, I increasingly felt like the goal of my work should not be to just create a pure functional effect wrapper for side-effects.</p>
<p>That had been done before, and while performance could be improved over Scalaz 7, libraries built on pure functional programming cannot generally compete with bare-metal, abstraction-free, hand-optimized procedural code.</p>
<p>Now, for developers like me who believe in the One True Way of functional programming, an effect-wrapper is enough, but it’s not going to win any new converts if it just slaps a Certified Pure label on imperative code. Although solving the async problem is still useful in a pre-Loom world, lots of other other data types already solved this problem, such as Scala’s own Future.</p>
<p>Instead, I thought I needed to focus the library on concrete pains that are well-solved by functional programming, and one pain stood out among all others: concurrency, including the unsolved problem in Scala’s Future of how to safely cancel executing code whose result is no longer necessary due to timeout or other failures.</p>
<p>Concurrency is a big enough space to play in that whole libraries and ecosystems have been formed to tame its wily ways. Indeed, some frameworks become popular precisely because they shield developers from concurrency, because it’s complex, confusing, and error-prone.</p>
<p>Given the challenges, I thought I should bake concurrency into this new data type, and give it features that would be impossible to replicate in a procedural program without special language features.</p>
<p>This vision of a powerful concurrent library drove my early development.</p>
<h2 id="contentious-concurrency">Contentious Concurrency</h2>
<p>At the time I began working on the Scalaz 8 project, the prevailing dogma in the functional Scala ecosystem was that an effect type should have little or no support for concurrency baked in.</p>
<p>Indeed, some <a href="https://web.archive.org/web/20200725224830/https://gist.github.com/djspiewak/a775b73804c581f4028fea2e98482b3c">argued that effect concurrency was inherently unsafe</a> and must be left to streaming libraries, like FS2.</p>
<p>Nonetheless, having seen some of the amazing work coming out of Haskell and F#, I believed it was not only possible but very important for a modern effect type to solve four closely related concurrency concerns:</p>
<ul>
<li>Starting a new independent ‘thread’ of computation.</li>
<li>Asynchronously “waiting” for a ‘thread’ to finish computing its return value.</li>
<li>Automatically canceling a running ‘thread’ when its return value is no longer needed.</li>
<li>Ensuring cancellation does not leak resources.</li>
</ul>
<p>In the course of time, I developed a small prototype of the so-called “Scalaz 8 IO monad”, which solved these problems in a fast and purely functional package: effects could be “forked” to yield a “fiber” (cooperatively-yielding virtual thread), which could be awaited or instantly interrupted, with a Haskell-inspired version of try/finally called bracket that provided resource safety.</p>
<p>I was excited about this design and talked about it publicly before I released the code, resulting in some backlash from competitors who didn’t believe the benchmarks could possibly be correct. But on November 16, 2017, <a href="https://www.youtube.com/watch?v=wi_vLNULh9Y&feature=emb_logo">I presented the first version</a> at Scale by the Bay, opening a pull request with full source code, including rigorously designed benchmarks, which allayed all concerns.</p>
<p>Despite initial skepticism and criticism, in time all effect systems in Scala adopted the concept of launching an effect to yield a Fiber, which could be safely and automatically interrupted, with a primitive to ensure resource safety.</p>
<p>This early prototype was not yet ZIO as we know it today, but the seeds of ZIO had been planted, and they grew quickly.</p>
<h2 id="typed-errors--other-evolution">Typed Errors & Other Evolution</h2>
<p>My initial design for the Scalaz 8 IO data type was inspired by Haskell and Scalaz Task. In due time, however, I found myself reevaluating some decisions made by these data types.</p>
<p>In particular, I was unhappy with the fact that most effect types have dynamically typed errors. Pragmatically speaking, the compiler can’t help you reason about error behavior if you pretend that every computation can always fail in infinite ways.</p>
<p>As a statically-typed functional programmer, I want to use the compiler to help me write better code. I’m able to do a better job if I know where I haven’t handled errors, and where I have, and if I can use typed data models for business errors.</p>
<p>Of course, Haskell an answer to adding statically-typed errors: use a monad transformer, and maybe type classes. Unfortunately, this solution increases barriers to entry, reduces performance, and bolts on a second, often confusing error channel.</p>
<p>Since I was starting from a clean slate in a different programming language, I had a different idea: what if instead of rigidly fixing the error type to <code class="language-plaintext highlighter-rouge">Throwable</code>, like the Scala <code class="language-plaintext highlighter-rouge">Try</code> data type, I let the user pick the type, like <code class="language-plaintext highlighter-rouge">Either</code>?</p>
<p>Initial results of my experiment were remarkable: just looking at type signatures, I could understand exactly how code was dealing with errors (or not dealing with them). Combinators precisely reflected error handling behavior in type signatures, and some laws that traditionally had to be checked using libraries like ScalaCheck were now checked statically, at compile time.</p>
<p>So on January 2, 2018, I committed what ended up being a radical departure from the status quo: <a href="https://github.com/scalaz/scalaz/commit/0d4766d7b375b4b94aa5bc9efe7772e2fcc1e982">introducing statically-typed errors</a> into the Scalaz 8 effect type.</p>
<p>Over the months that followed, I worked on polish, optimization, bug fixes, unit tests, and documentation, and found growing demand to use the data type in production. When it became apparent that Scalaz 8 was a longer-term project, a few ambitious developers pulled the IO data type into a <a href="https://github.com/scalaz/ioeffect">standalone library</a> so they could begin using it in their projects.</p>
<p>I was excited about this early traction, and I didn’t want any obstacles to using the data type for users of Scalaz 7.x or Cats, so on June 11, 2018, I decided to pull the project out into a new, standalone project with zero dependencies, completely separate from Scalaz 8. I chose the name “ZIO”, combining the “Z” from “Scalaz”, and the “IO” from “IO monad”.</p>
<h2 id="contributor-growth">Contributor Growth</h2>
<p>Around this time, the first significant wave of contributors started joining the project, including Regis Kuckaertz, Wiem Zine Elabidine, and Pierre Ricadat (among others)—many new to both open source and functional Scala, although some with deep backgrounds in both.</p>
<p>Through mentorship by me and other contributors, including in some cases weekly meetings and pull request reviews, a whole new generation of open source Scala contributors were born—highly talented functional Scala developers whose warmth, positivity and can-do spirit started to shape the community.</p>
<p>ZIO started accreting more features to make it easier to build concurrent applications, such as an asynchronous, doubly-back-pressured queue, better error tracking and handling, rigorous finalization, and lower-level resource-safety than bracket.</p>
<p>Although the increased contributions led to an increasingly capable effect type, I personally found that using ZIO was not very pleasant, because of the library’s poor type inference.</p>
<h2 id="improved-type-inference">Improved Type-Inference</h2>
<p>As many functional Scala developers at the time, I had absorbed the prevailing wisdom about how functional programming should be done in Scala, and this meant avoiding subtyping and declaration-site variance (indeed, the presence of subtyping in a language does negatively impact type inference, and using declaration-site variance has a couple drawbacks).</p>
<p>However, because of this decision, using ZIO required specifying type parameters to many methods, resulting in an unforgiving and joyless style of programming, particularly with typed errors. In private, I wrote a small prototype showing that using declaration-site variance could significantly improve type inference, which made me want to implement the feature in ZIO.</p>
<p>At the time, however, ZIO still resided in the Scalaz organization, in a separate repository. I was aware that such a departure from the status quo would be controversial, so in a private fork, Wiem Zine Elabidine and I worked together on a massive refactoring in our first major collaboration.</p>
<p>On Friday July 20, 2018, we opened the pull request that added covariance. The results spoke for themselves: nearly all explicit type annotations were deleted, and although there was some expected controversy, it was difficult to argue with the results. With this change, ZIO started becoming pleasant to use, and the extra error type parameter no longer negatively impacted usability.</p>
<p>This experience emboldened me to start breaking other taboos: I started aggressively renaming methods and classes and removing jargon known only to pure functional programmers. At each step, this created yet more controversy, but also further differentiated ZIO from some of the other design choices, including those expressed in Scalaz 7.x.</p>
<p>From all this turbulent evolution, a new take on functional Scala entered the ZIO community: a contrarian but principled take that emphasizes practical concerns, solving real problems in an accessible and joyful way, using all parts of Scala, including subtyping and declaration-site variance.</p>
<p>Finally, the project began to feel like the ZIO of today, shaped by a rapidly growing community of fresh faces eager to build a new future for functional programming in Scala.</p>
<h2 id="batteries-included">Batteries Included</h2>
<p>Toward the latter half of 2018, ZIO got compositional scheduling, with a powerful new data type that represents a schedule, equipped with rich compositional operators. Using this single data type, ZIO could either retry effects or repeat them according to near arbitrary schedules.</p>
<p>Artem Pyanykh implemented a blazing fast low-level ring-buffer, which, with the help of Pierre Ricadat, became the foundation of ZIO’s asynchronous queue, demonstrating the ability of the ZIO ecosystem to create de novo high-performance JVM structures.</p>
<p>Itamar Ravid, a highly talented Scala developer, joined the ZIO project and added a Managed data type encapsulating resources. Inspired by Haskell, Managed provided compositional resource safety in a package that supported parallelism and safe interruption. With the help of Maxim Schuwalow, Managed has grown to become an extremely powerful data type.</p>
<p>Thanks to the efforts of Raas Ahsan, ZIO unexpectedly got an early version of what would later become <code class="language-plaintext highlighter-rouge">FiberRef</code>, a fiber-based version of <code class="language-plaintext highlighter-rouge">ThreadLocal</code>. Then Kai, a wizard-level Scala developer and type astronaut, labored to add compatibility with Cats Effect libraries, so that ZIO users could benefit from all the hard work put into libraries like Doobie, http4s, and FS2.</p>
<p>Thanks to the work of numerous contributors spread over more than a year, ZIO became a powerful solution to building concurrent applications—albeit, one without concurrent streams.</p>
<h2 id="zio-stream">ZIO Stream</h2>
<p>Although Akka Streams provides a powerful streaming solution for Scala developers, it’s coupled to the Akka ecosystem and Scala’s Future. In the functional Scala space, FS2 provides a streaming solution that works with ZIO, but it’s based on Cats Effect, whose type classes can’t benefit from ZIO-specific features.</p>
<p>I knew that a ZIO-specific streaming solution would be more expressive and more type safe, with a lower barrier of entry for existing ZIO users. Given the importance of streaming to modern applications, I decided that ZIO needed its own streaming solution, one unconstrained by the feature set of Cats Effect.</p>
<p>Bringing a new competitive streaming library into existence would be a lot of work, and so when Itamar Ravid volunteered to help, I instantly said yes.</p>
<p>Together, in the third quarter of 2018, Itamar and I worked in secret on ZIO Streams, an asynchronous, back-pressured, resource-safe, and compositional stream. Inspired by work that the remarkable Scala developer Eric Torreborre did, as well as work in Haskell on iteratees, the initial release of ZIO Streams delivered a high-performance, composable concurrent streams and sinks, with strong guarantees of resource safety, even in the presence of arbitrary interruption.</p>
<p>We unveiled the design at <a href="https://www.youtube.com/watch?v=mLJYODobz44">Scale by the Bay</a> 2018, and since then, thanks to Itamar and his army of capable contributors (including Regis Kuckaertz), ZIO Streams has become one of the highlights of the ZIO library—every bit as capable as other streaming libraries, but with much smoother integration with the ZIO effect type and capabilities.</p>
<p>Toward the end of 2018, I decided to focus on the complexity of testing code written using effect systems, which led to the last major revision of the ZIO effect type.</p>
<h2 id="zio-environment">ZIO Environment</h2>
<p>When exploring a contravariant reader data type to model dependencies, I discovered that using intersection types (emulated by the with keyword in Scala 2.x), one could achieve flawless type inference when composing effects with different dependencies, which provided a possible solution to simplifying testing of ZIO applications.</p>
<p>Excitedly, I wrote up a simple toy prototype and shared it with Wiem Zine Elabidine. <em>“Do you want to help work on this?”</em> I asked. She said yes, and together we quietly added the third and final type parameter to the ZIO effect type: the environment type parameter.</p>
<p>I unveiled the third type parameter at a now-infamous talk, The Death of Finally Tagless, humorously presented with a cartoonish Halloween theme. In this talk, I argued that testability was the primary benefit of tagless-final, and that it could be obtained much more simply and in a more teachable way by just “passing interfaces”—the solution that object-oriented programmers have used for decades.</p>
<p>As with tagless-final, and under the assumption of discipline, ZIO Environment provided a way to reason about dependencies statically, and a way to code to interfaces without any pain at the use-site. Unlike tagless-final, it fully inferred and didn’t require teaching type classes, category theory, higher-kinded types, or implicits.</p>
<p>Some ZIO users immediately started using ZIO Environment, appreciating the ability to describe dependencies using types without actually passing them (thus allowing dependency inference). Constructing ZIO environments, however, proved to be problematic—impossible to do generically, and difficult to do even when the structure of the environment was fully known.</p>
<p>A workable solution to these pains would not be identified until almost a year later.</p>
<p>Meanwhile, ZIO continued to benefit from numerous contributions, which added combinators, improved documentation, improved interop, and improved semantics for core data types.</p>
<p>The next major addition to ZIO was software transactional memory.</p>
<h2 id="software-transactional-memory">Software Transactional Memory</h2>
<p>The first prototype of the Scalaz IO data type included <code class="language-plaintext highlighter-rouge">MVar</code>, a doubly-back-pressured queue with a maximum capacity of 1, inspired by Haskell’s data type of the same name.</p>
<p>I really liked the fact that MVar was already “proven”, and could be used to build many other concurrent data structures (such as queues, semaphores, and more).</p>
<p>Soon after that early prototype, however, the talented and eloquent Fabio Labella convinced me that two simpler primitives provided a more orthogonal basis for building concurrency structures:</p>
<ul>
<li>Promise, a variable data type that can be set exactly one time (but can be awaited on asynchronously and retrieved any number of times);</li>
<li>Ref, a model of a mutable cell that can store any immutable value, with atomic operations for updating the cell.</li>
</ul>
<p>This early refactoring allowed us to delete <code class="language-plaintext highlighter-rouge">MVar</code> and provided a much simpler foundation. However, after a year of using these structures, while I appreciated their orthogonality and power, it became apparent to me that they were the “assembly language” of concurrent data structures.</p>
<p>These structures could be used to build lots of other asynchronous concurrent data structures, such as semaphores, queues, and locks, but doing so was extremely tricky, and required hundreds of lines of fairly advanced code.</p>
<p>Most of the complexity stems from the requirement that operations on the data structures must be safely interruptible, without leaking resources or “deadlocking”.</p>
<p>Moreover, although you can build concurrent structures with <code class="language-plaintext highlighter-rouge">Promise</code> and <code class="language-plaintext highlighter-rouge">Ref</code> with enough work, you cannot make coordinated changes across two or more such concurrent structures.</p>
<p>The transactional guarantees of structures built with <code class="language-plaintext highlighter-rouge">Promise</code> and <code class="language-plaintext highlighter-rouge">Ref</code> are non-compositional: they apply only to isolated data structures, because they are built with <code class="language-plaintext highlighter-rouge">Ref</code>, which has non-compositional transactional semantics. Strictly speaking, their transactional power is equivalent to actors with mutable state: each actor can safely mutate its own state, but no transactional changes can be made across multiple actors.</p>
<p>Familiar with Haskell’s software transactional memory, and how it provides an elegant, compositional solution to the problem of developing concurrent structures, I decided to implement a version for ZIO with the help of my partner-in-crime Wiem Zine Elabidine, which we presented at Scalar Conf in April 2019.</p>
<p>Dejan Mijic, another fantastic and highly motivated developer with a keen interest in high-performance, concurrency, and distributed systems, joined the ZIO STM team. With my mentorship, Dejan helped make STM stack-safe for transactions of any size, added several new STM data structures, dramatically improved the performance of existing structures, and implemented retry-storm protection for supporting large transactions on hotly contested transactional references.</p>
<p>ZIO STM is the only STM in Scala with these features, and although the much older Scala STM is surely production-worthy, it doesn’t integrate well with asynchronous and purely functional effect systems built using fiber-based concurrency.</p>
<p>The next major feature in ZIO would address a severe deficiency that had never been solved in the Scala ecosystem: the extreme difficulty of debugging async code, a problem present in Scala’s Future for more than a decade.</p>
<h2 id="execution-traces">Execution Traces</h2>
<p>Previously in presenting ZIO to new non-pure functional programmers (the primary audience for ZIO), I had received the question: how do we debug ZIO code?</p>
<p>The difficulty stems from the worthless nature of stack traces in highly asynchronous programming. Stack traces only capture the call stack, but in Future and ZIO and other heavily async environments, the call stack mainly shows you the “guts” of the execution environment, which is not very useful for tracking down errors.</p>
<p>I had thought about the problem and had become convinced it would be possible to implement async execution traces using information reconstructed from the call stack, so I began telling people we would implement something like this in ZIO soon.</p>
<p>I did not anticipate just how soon this would be.</p>
<p>Kai came to me with an idea to do execution tracing in a radically different way than I imagined: by dynamically parsing and executing the bytecode of class files. Although my recollection is a bit hazy, it seemed mere days before Kai had whipped up a prototype that seemed extremely promising, so I offered my assistance on hammering out the details of the full implementation, and we ended up doing a wonderful joint talk in Ireland to launch the feature.</p>
<p>Sometimes I have a tendency to focus on laws and abstractions, but seeing the phenomenally positive response to execution tracing was a good reminder to stay focused on the real world pains that developers have.</p>
<h2 id="summer-2019">Summer 2019</h2>
<p>Beginning in the summer of 2019, ZIO began seeing its first significant commercial adoption, which led to many feature requests and bug reports, and much feedback from users.</p>
<p>The summer saw many performance improvements, bug fixes, naming improvements, and other tweaks to the library, thanks to Regis Kuckaertz and countless other contributors.</p>
<p>Thanks to the work of the ever-patient Honza Strnad and others, <code class="language-plaintext highlighter-rouge">FiberRef</code> evolved into its present-day form, which is a much more powerful, fiber-aware version of ThreadLocal—but one which can undergo specified transformations on forks, and merges on joins.</p>
<p>I was very pleased with these additions. However, as ZIO grew, the automated tests for ZIO were growing too, and they became an increasing source of pain across Scala.js, JVM, and Dotty (our test runners at the time did not natively support Dotty).</p>
<p>So in the summer of 2019, I began work on a purely functional testing framework, with the goal of addressing these pains, the result of which was ZIO Test.</p>
<h2 id="zio-test">ZIO Test</h2>
<p>Testing functional effects inside a traditional testing library is painful: there’s no easy way to run effects, provide them with dependencies, or integrate with the host facilities of the functional effect system (using retries, repeats, and so forth).</p>
<p>I wanted to change that with a small, compositional library called ZIO Test, whose design I had been thinking about since even before ZIO existed.</p>
<p>Like the ground-breaking Specs2 before it, ZIO Test embraced a philosophy of tests as values, although ZIO Test retained a more traditional tree-like structure for specs, which allows nesting tests inside test suites, and suites inside other suites.</p>
<p>Early in the development of ZIO Test, the incredible and extremely helpful Adam Fraser joined the project as a core contributor. Instrumental to fleshing out, realizing, and greatly extending the vision for ZIO Test, Adam has since become the lead architect and maintainer for the project.</p>
<p>Piggybacking atop ZIO’s powerful effect type, ZIO Test was implemented in comparatively few lines of code: concerns like retrying, repeating, composition, parallel execution, and so forth, were already implemented in a principled, performant, and type-safe way.</p>
<p>Indeed, ZIO Test also got a featherweight alternative to ScalaCheck based on ZIO Streams, since a generator of a value can be viewed as a stream. Unlike ScalaCheck, the ZIO Test generator has auto-shrinking baked in, inspired by the Haskell Hedgehog library; and it correctly handles filters on shrunk values and other edge case scenarios that ScalaCheck did not handle.</p>
<p>Toward the end of 2018, after nearly a year of real world usage, the ZIO community had been hard at work on solutions to the problem of making dynamic construction of ZIO environments easier.</p>
<p>This work directly led to the creation of ZLayer, the last major data type added to ZIO.</p>
<h2 id="zlayer">ZLayer</h2>
<p>Two very talented Scala developers, Maxim Schuwalow and Piotr Gołębiewski, jointly worked on a ZIO Macros project, which, among other utilities, provided an easier way to construct larger ZIO environments from smaller pieces. This excellent work was independently replicated in the highly-acclaimed Polynote by Jeremy Smith in response to the same pain.</p>
<p>At Functional Scala 2019, several speakers presented on the pain of constructing ZIO Environments, which convinced me to take a hard look at the problem. Taking inspiration from an earlier attempt by Piotr, I created two new data types, <code class="language-plaintext highlighter-rouge">Has</code> and <code class="language-plaintext highlighter-rouge">ZLayer</code>.</p>
<p><code class="language-plaintext highlighter-rouge">Has</code> can be thought of as a type-indexed heterogeneous map, which is type safe, but requires access to compile-time type tag information. <code class="language-plaintext highlighter-rouge">ZLayer</code> can be thought of as a more powerful constructor, which can build multiple services in terms of their dependencies.</p>
<p>Unlike constructors, ZLayer dependency graphs are ordinary values, built from other values using composable operators, and ZLayer supports resources, asynchronous creation and finalization, retrying, and other features not possible with constructors.</p>
<p>ZLayer provided a very clean solution to the problems developers were having with ZIO Environment—not perfect, mind you, and I don’t think any solution prior to Scala 3 can be perfect (every solution in the design space has different tradeoffs). The good solution became even better when the excellent consultancy Septimal Mind donated Izumi Reflect to the ZIO organization.</p>
<p>The introduction of ZLayer was the last major change to any core data type in ZIO. Since then, although streams has seen some churn, the rest of ZIO has been very stable.</p>
<p>Yet despite the stability, until recently, there was still one major unresolved issue at the very heart of the ZIO runtime system: a full solution to the problem of structured concurrency.</p>
<h2 id="structured-concurrency">Structured Concurrency</h2>
<p>Structured concurrency is a paradigm that provides strong guarantees around the lifespans of operations performed concurrently. These guarantees make it easier to build applications that have stable, predictable resource utilization.</p>
<p>Since I have long been a fan of Haskell structured concurrency (via Async and related), ZIO was the first effect system to support structured concurrency in numerous operations:</p>
<ul>
<li>By default, interrupting a fiber does not return until the fiber has been interrupted and all its finalizers executed.</li>
<li>By default, timing out an effect does not return until the effect being timed out has been interrupted and all its finalizers executed.</li>
<li>By default, when executing effects in parallel, if one of them fails, the parallel operation will not continue until all sibling effects have been interrupted.</li>
<li>Etc.</li>
</ul>
<p>Some of these design decisions were contentious and have not been implemented in other effect systems until recently (if at all).</p>
<p>However, there was one notable area where ZIO did not provide structured concurrency by default: whenever an effect was forked (launched concurrently to execute on a new fiber), the lifespan of that executing effect was unconstrained.</p>
<p>Solving this problem turned out to require multiple major surgeries to ZIO’s internal runtime system (which is a part of ZIO that few developers understand completely, and which tends to bottleneck on me).</p>
<p>In the end, we solved the problem in a satisfactory way, but it required learning from real world feedback and prototyping no less than 5 completely different solutions to the problem.</p>
<p>Today, ZIO is the only effect system in Scala with full structured concurrency by default.</p>
<h2 id="zio-10--beyond">ZIO 1.0 & Beyond</h2>
<p>Yesterday, on August 3rd, ZIO 1.0 was released live in an online Zoom-hosted launch party that brought together and paid tribute to contributors and users across the ZIO ecosystem. We laughed, we chatted, I monologued a bit, and we toasted a few times to users, contributors, and the past and future of ZIO.</p>
<p>As of today, the ZIO 1.0 artifacts are now available on Sonatype, and other parts of the ZIO ecosystem are working to rapidly release new versions of downstream libraries.</p>
<p>We expect full binary backward compatibility for the 1.x line, with two exceptions:</p>
<ul>
<li>ZIO Streams. We will not lock down the ZIO Streams API until ZIO 1.1 (or higher), allowing us more time to prove out some newer improvements to the library.</li>
<li>ZIO Test. Similarly to ZIO Streams, we want to wait until ZIO 1.1 at least before locking down ZIO Test. While no major breaking changes are planned, we need to solve the problem of shared layers across totally different specs, as well as reporting, and it could be in solving these problems, we need to make a few breaking changes.</li>
</ul>
<p>These guarantees of backward compatibility will help the ZIO ecosystem flourish, and encourage more corporate adoption.</p>
<p>The <a href="https://github.com/zio/zio/releases/tag/v1.0.0">release notes for 1.0 are online</a>.</p>
<h2 id="why-zio">Why ZIO</h2>
<p>Effect systems aren’t right for everyone, and ZIO may not be the right choice for some teams, but I think there are compelling reasons for Scala developers to take a serious look at ZIO 1.0:</p>
<ul>
<li><strong>Wonderful Community</strong>. I’m amazed at all the talent, positivity, and apprenticeship seen in the ZIO community, as well as the universal attitude that we are all on the same page, there is no your code, or my code, just our code, and we will collaboratively work together on ways of improving our code.
<ol>
<li><strong>History of Innovation</strong>. ZIO was the first effect type with pure, automatic interruption on by default; the first effect type with execution context locking that correctly handled auto-shifting on async boundaries and exceptions; the first effect type with typed errors and the only effect type with environment; the first effect type with execution tracing; the only current effect type with fine-grained interruption; the only current effect type with full structured concurrency; and so much more. Although inspired by Haskell, ZIO has charted its own course and become what many believe to be the leading effect system in Scala.</li>
<li><strong>A Bright Future</strong>. ZIO took three years to get right, because I believed the foundations had to be extremely robust to support what was coming. And if early libraries like Caliban, ZIO Redis, ZIO Config, ZIO gRPC, and others are any indication, what is coming is going to be amazing for ZIO users and attract new users both to functional programming, and to the Scala programming language itself. In addition, after living with an advanced effect system for the past 3 years, we core contributors have some very exciting ideas about what’s next in the future of effect systems, and several of these are mind-blowing.</li>
</ol>
</li>
</ul>
<p>Whether you end up with ZIO or something else, there’s no question that concurrent programming in Scala was never this much fun, correct-by-construction, or productive!</p>
<h2 id="personal-thanks">Personal Thanks</h2>
<p>ZIO 1.0 would not be possible without the work of more than 316 contributors. I personally reviewed and thanked many of these contributors, but I want to thank each and every one of them now for their work, their creativity, and their inspiration.</p>
<p>In addition, I’d like to personally thank some of the core contributors, who have not only rolled up their sleeves and contributed directly, but have mentored other developers, given talks, inspired me, and contributed an incredible variety of creative suggestions, innovative features, bug fixes, automated tests, and documentation improvements.</p>
<p>In particular, I want to thank:</p>
<ul>
<li><em>Wiem Zine Elabidine</em>, for being the first person to believe in the project and for having such an enormous role in co-developing what ZIO would become; and for being my friend and ongoing source of inspiration, in good times and in bad.</li>
<li><em>Adam Fraser</em>, for being, literally, my right hand man, for being the top contributor to ZIO (congratulations!), for continuously innovating in ways that never cease to amaze me, and for being one of the most helpful and impactful voices in the ZIO community.</li>
<li><em>Itamar Ravid</em>, for having the depth and breadth of experience that comes with seniority, and being able to tackle any problem whatsoever in any part of the code base; you’re a machine (and incidentally, the first person I’d choose to go to war with…or hit the gym with, whatever!), and it’s a pleasure to have you as co-contributor and colleague;</li>
<li><em>Pierre Ricadat</em>, for being one of the lights that shines so brightly across Scala, with not only contributions to ZIO, but the creation of the amazing Caliban library, which shows everyone how powerful and joyful functional Scala can be; and for mentoring new generations of functional Scala developers and producing the community’s wonderful newsletter;</li>
<li><em>Dejan Mijić</em>, for being a wonderful friend and co-conspirator in a never-ending stream of improvements (many of which take place ‘concurrently’), for taking on even the craziest of performance challenges with glee, and for leaving me with fun memories of it all;</li>
<li><em>Regis Kuckaertz</em>, for being one of the brave people to join me on the journey in the early days, working through some of the trickiest code in the ZIO runtime system, and for writing the most entertaining documentation I’ve seen in any OSS project;</li>
<li><em>Kai</em>, for being Kai—a one of a kind, force of nature who goes where others fear to tread, usually innovating on multiple fronts simultaneously (because one is never enough!), and telling me things I may not like to hear, but definitely should hear;</li>
<li><em>Artem Pyanykh</em>, for blowing me away with attention to detail and some of the finest system-level code I’ve seen in the Scala ecosystem;</li>
<li><em>Salar Rahmanian</em>, Tim Steinbach, for always pushing our build and release process to the incredible machine it has become, and for saving me on many occasion when things were broken and I didn’t understand what was going on;</li>
<li><em>Lech Głowiak</em>, for diving into both functional Scala and open source development, helping wherever needed, despite an impossibly busy life and work schedule; </li>
<li><em>Maxim Schuwalow</em> & <em>Piotr Gołębiewski</em>, for their ongoing independent innovation that has significantly shaped 1.0, and for always pushing to find better ways of solving the problems ZIO addresses;</li>
<li><em>Igal Tabachnik</em>, for building the most amazing IntelliJ Plugin I’ve seen in the Scala ecosystem, providing a level of comfort and maturity that Java users receive;</li>
<li><em>Francois Armand</em>, for being the constant voice of the user, giving gentle feedback on ways we can make the project deal with the very real pains that users have (yes, Francois, auto-blocking is now officially on the TODO list!).</li>
<li><em>Flavio Brasil</em>, for his inspirational body of work at the intersection of performance and functional programming, and although Flavio has not yet been able to contribute to ZIO directly, his encouragement to work on super fast collections led to Chunk, which is a box-free ZIO data type backed by arrays with super fast element append/prepend, great cache locality, and O(1) with small constant factors for most operations.</li>
</ul>
<p>I want to thank all the early adopters of ZIO, who fearlessly rolled out ZIO into production since the 0.22 release, and who were willing to pay the tax of tirelessly updating their code base, sometimes with each new release candidate for ZIO. These adopters encountered problems, reported bugs, and proposed new features that helped shape ZIO into the library it has become.</p>
<p>I want to thank <a href="https://7mind.io/">Septimal Mind</a>, who donated Izumi Reflect and contributed execution tracing; <a href="https://softwaremill.com/">SoftwareMill</a>, for their financial support for CircleCI builds and for being outstanding community supporters; <a href="https://scalac.io/">Scalac</a>, for their ongoing work teaching ZIO, sponsorship of ZIO Hackathon Warsaw, and their contributions to multiple ZIO open source projects (including the amazing <a href="https://github.com/ScalaConsultants/panopticon-tui">panopticon-tui</a>); and <a href="https://www.signifytechnology.com/">Signify Technology</a>, who has helped bring many ZIO jobs to Scala developers, and whose participation in the Scala community has greatly enriched it.</p>
<p>Finally, I want to acknowledge the Haskell ecosystem, in particular Simon Marlow, who pushed the boundaries of what’s possible in functional concurrency and asynchronicity; Michael Snoyman, who invented RIO, which shares a common design and purpose with ZIO; Michael Pilquist, who came up with a brilliant trick for making asynchronous waiting in concurrent structures safely interruptible (we used this in ZIO for the original version of Semaphore, though it’s not necessary with STM, which provides this property for free); Alex Nedelcu, whose work on Monix blew past Scalaz Task long before there was ZIO, and whose work inspired a wonderful peek-ahead optimization in ZIO’s runtime; and Fabio Labella, who persuaded me to abandon <code class="language-plaintext highlighter-rouge">MVar</code> in favor of more orthogonal primitives.</p>
<p>This list could never be complete, and if I have omitted any significant credit or contribution, it’s only because I’m human.</p>
<p>This has been an amazing journey for me, one I can scarcely believe has concluded. I’m incredibly grateful for the experience and the opportunity to finally release of ZIO 1.0, and start a new journey to places unknown.</p>
<p><a href="https://degoes.net/articles/zio-1.0">ZIO 1.0 Released</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on August 03, 2020.</p>https://degoes.net/articles/no-effect-tracking2020-05-03T00:00:00-06:002020-05-03T00:00:00-06:00John A De Goeshttps://degoes.netjohn@degoes.net<p>Effect tracking is not a valid reason to use functional effect systems, because effect tracking is commercially worthless.</p>
<p>More precisely, companies that pay their software developers to “track effects” will not obtain a return on this investment, but rather, will lose money.</p>
<p>In this post, I’ll explain why.</p>
<h2 id="side-effects">Side Effects</h2>
<p>The whole notion of “effects” doesn’t make a lot of sense outside <em>functional programming</em>.</p>
<p>In functional programming, we try to build our software from deterministic, pure functions. These functions are like the ones most of us learned about in high school:</p>
<pre>
f(x) = x * x
</pre>
<p>Such functions are said to be “free of side-effects”, meaning their only effect is combining and transforming inputs to produce an output.</p>
<p>Pure functions don’t do anything <em>on the side</em>—they don’t do anything that could be observed from outside the function, like operating system calls or mutating the heap.</p>
<h2 id="discovering-purity">Discovering Purity</h2>
<p>When developers first discover functional programming, they believe that it’s <em>useless</em> for real world applications.</p>
<p>I had the same response when I first heard about programming without procedural statements, assignments, or loops.</p>
<p>After all, procedures that compute random numbers, call databases, and or invoke web APIs, all perform “side-effects”.</p>
<p>These procedures are not deterministic, pure functions that just combine and transform inputs to produce outputs. Yet they are the <em>cornerstone</em> of building applications that solve real business problems.</p>
<p>Eventually, of course, I discovered that in purely functional languages like Haskell, functions that interact with the outside world all return values of a mysterious <code class="language-plaintext highlighter-rouge">IO</code> data type.</p>
<p>This discovery feeds a myth that’s pervasive among even experienced Haskell programmers.</p>
<p>The myth of <em>effect tracking</em>.</p>
<h2 id="effect-tracking">Effect Tracking</h2>
<p>It’s extraordinarily common for developers who have encountered Haskell’s <code class="language-plaintext highlighter-rouge">IO</code> data type to explain it as follows:</p>
<ul>
<li>Impure functions return <code class="language-plaintext highlighter-rouge">IO</code></li>
<li>Pure functions don’t return <code class="language-plaintext highlighter-rouge">IO</code></li>
</ul>
<p>According to this explanation, one can determine whether or not a function is effectful merely by examining the return type. The <code class="language-plaintext highlighter-rouge">IO</code> type therefore “tracks” the presence of side-effects… hence, <em>effect tracking</em>.</p>
<p>Consequently, many developers now believe that Haskell uses the <code class="language-plaintext highlighter-rouge">IO</code> type to track effects, to mark them as being “impure”—that Haskell’s type system is being recruited to help us ascertain purity.</p>
<p>There’s just one <em>tiny</em> little problem with this explanation.</p>
<p>It’s totally and completely wrong!</p>
<h2 id="myth-busting">Myth Busting</h2>
<p>In point of fact, a Haskell function that returns <code class="language-plaintext highlighter-rouge">IO</code> <strong>is</strong> pure: it’s deterministic and entirely free of side-effects, merely transforming and combining inputs to produce an output.</p>
<p>For example, take the <code class="language-plaintext highlighter-rouge">putStrLn</code> function:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">putStrLn</span> <span class="o">::</span> <span class="kt">String</span> <span class="o">-></span> <span class="kt">IO</span> <span class="nb">()</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">putStrLn</span><span class="o">(</span><span class="n">line</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">IOException</span>, <span class="kt">Unit</span><span class="o">]</span></code></pre></figure>
<p>This function takes a string, and returns an <code class="language-plaintext highlighter-rouge">IO</code> value. It does not actually <em>perform</em> any effects, and if given the same string, it will return the ‘same’ <code class="language-plaintext highlighter-rouge">IO</code> value. Moreover, the value that it returns is <em>totally immutable</em>, like all values in Haskell.</p>
<p>These guarantees hold not just for <code class="language-plaintext highlighter-rouge">putStrLn</code>, but for all functions that return <code class="language-plaintext highlighter-rouge">IO</code> values (unless they cheat by using <code class="language-plaintext highlighter-rouge">unsafePerformIO</code> or similar).</p>
<p>Stated more forcefully, functions that return <code class="language-plaintext highlighter-rouge">IO</code> values are <em>no different</em> than functions that do <strong>not</strong> return <code class="language-plaintext highlighter-rouge">IO</code> values: they are pure, and their type signature does not reveal the presence or absence of side-effects, because there are no side-effects at all.</p>
<p>Now, eventually one learns that <code class="language-plaintext highlighter-rouge">IO</code> is a data type, whose immutable values <em>model</em> some sequence of interactions with the outside world (mutable heap, sockets, files, databases, and the like).</p>
<p>So, even though a function like <code class="language-plaintext highlighter-rouge">putStrLn</code> does not <em>perform</em> any side-effects, the value it returns <em>describes</em> a side-effect.</p>
<p>However, <code class="language-plaintext highlighter-rouge">IO</code> is not magical in this regard. Indeed, entirely without <code class="language-plaintext highlighter-rouge">IO</code>, we can easily construct a simple model of interaction with the outside world, like sockets:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="kr">data</span> <span class="kt">SocketIO</span> <span class="n">a</span> <span class="o">=</span>
<span class="kt">ReadByte</span> <span class="p">(</span><span class="kt">Byte</span> <span class="o">-></span> <span class="kt">SocketIO</span> <span class="n">a</span><span class="p">)</span> <span class="o">|</span>
<span class="kt">WriteByte</span> <span class="kt">Byte</span> <span class="p">(</span><span class="kt">SocketIO</span> <span class="n">a</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">SocketIO</span><span class="o">[</span><span class="kt">+A</span><span class="o">]</span>
<span class="nc">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">ReadByte</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">withByteDo</span><span class="k">:</span> <span class="kt">Byte</span> <span class="o">=></span> <span class="nc">SocketIO</span><span class="o">[</span><span class="kt">A</span><span class="o">])</span> <span class="k">extends</span> <span class="nc">SocketIO</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span>
<span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">WriteByte</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">byte</span><span class="k">:</span> <span class="kt">Byte</span><span class="o">,</span> <span class="n">andThenDo</span><span class="k">:</span> <span class="kt">SocketIO</span><span class="o">[</span><span class="kt">A</span><span class="o">])</span> <span class="k">extends</span> <span class="nc">SocketIO</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span></code></pre></figure>
<p>Now one can write functions that return <code class="language-plaintext highlighter-rouge">SocketIO</code> values.</p>
<p><code class="language-plaintext highlighter-rouge">IO</code> values are no less pure than <code class="language-plaintext highlighter-rouge">SocketIO</code> values—literally the only distinction between them is that, in Haskell, your main function may return an <code class="language-plaintext highlighter-rouge">IO</code> value, and if it does, the data structure will be translated by the Haskell runtime into the side-effects that it models.</p>
<p>But we could easily do that ourselves for <code class="language-plaintext highlighter-rouge">SocketIO</code>, by having an interpreter that is built on <code class="language-plaintext highlighter-rouge">unsafePerformIO</code>!</p>
<h2 id="effect-tracking-1">Effect Tracking?</h2>
<p>So effect tracking—the ability to “track effects” in the type system—is a gigantic misnomer, because the <code class="language-plaintext highlighter-rouge">IO</code> type does not track side-effects.</p>
<p>There are no side-effects in Haskell programs (<code class="language-plaintext highlighter-rouge">unsafePerformIO</code> and friends notwithstanding).</p>
<p>There are only data types, some of which model interaction with the outside world. However, even a list of bytes can model interaction with the outside world. In fact, a binary program is nothing more than a list of bytes, which models interaction with the outside world using machine code instruction sets!</p>
<p>That said, if we ignore the technical sense in which there is no such thing as “effect tracking” in Haskell, can we say that having “tracked effects” is a good reason to use data types like <code class="language-plaintext highlighter-rouge">IO</code>?</p>
<p>The answer is no, because effect tracking can be done without <code class="language-plaintext highlighter-rouge">IO</code>!</p>
<h2 id="effect-tracked-java">Effect-Tracked Java™</h2>
<p>If, by <em>effect tracking</em>, we mean that we can know, looking only at the signature of a function, whether or not the method models or performs side-effects, then allow me to introduce to you <em>Effect-Tracked Java™</em>!</p>
<p>In Effect-Tracked Java™ (a feasible thought experiment for now), every method is annotated with either <code class="language-plaintext highlighter-rouge">@Pure</code> or <code class="language-plaintext highlighter-rouge">@Impure</code>. For example:</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@Impure</span>
<span class="kd">public</span> <span class="nc">Config</span> <span class="nf">loadAppConfig</span><span class="o">()</span> <span class="o">{</span>
<span class="o">...</span>
<span class="o">}</span>
<span class="nd">@Pure</span>
<span class="kd">public</span> <span class="nc">Config</span> <span class="nf">mergeConfigs</span><span class="o">(</span><span class="nl">defaults:</span> <span class="nc">Config</span><span class="o">,</span> <span class="nl">priority:</span> <span class="nc">Config</span><span class="o">)</span> <span class="o">{</span>
<span class="o">...</span>
<span class="o">}</span></code></pre></figure>
<p>Let’s assume that all the “root” methods in the Java standard library, together with JNI methods, have been ascribed these annotations, which are stored in a data file accessible to our build.</p>
<p>This information allows a “purity” annotation processor, which we would run as part of our build process, to verify that all our annotations are correct.</p>
<p>For example, if we call <code class="language-plaintext highlighter-rouge">System.currentTimeMillis()</code> inside a function annotated <code class="language-plaintext highlighter-rouge">@Pure</code>, we will get an error, and we will have to fix the error by changing the annotation to <code class="language-plaintext highlighter-rouge">@Impure</code>.</p>
<p>Presto, instant effect tracking, without esoteric “monads” or any of that pesky functional programming!</p>
<p>Further, we don’t have to stop at such coarse-grained effect tracking. If we like, we can allow <code class="language-plaintext highlighter-rouge">@Impure</code> annotations to introduce labels, which we could mistakenly call “algebras”.</p>
<p>This would give us (let’s call it) <em>Tagless-Final Effect-Tracked Java™</em>, which would look something like this:</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@Impure</span><span class="o">({</span><span class="s">"FileSystemAlgebra"</span><span class="o">,</span> <span class="s">"EnvironmentVariablesAlgebra"</span><span class="o">})</span>
<span class="kd">public</span> <span class="nc">Config</span> <span class="nf">loadAppConfig</span><span class="o">()</span> <span class="o">{</span>
<span class="o">...</span>
<span class="o">}</span></code></pre></figure>
<p>As before, our annotation processor would enforce correctness. So if we accidentally called <code class="language-plaintext highlighter-rouge">System.currentTimeMillis()</code> inside a function annotated <code class="language-plaintext highlighter-rouge">@Impure()</code>, we would be forced to change this annotation to <code class="language-plaintext highlighter-rouge">@Impure({"ClockAlgebra"})</code>.</p>
<p>With these “improvements”, we could tell not only which methods are pure and impure, but for the impure methods, we could tell which “algebras” they use—and all of this is tracked statically!</p>
<p>Have we just discovered a way to radically boost developer productivity for all Java developers everywhere?!?</p>
<p>Read on for my unexpected and totally surprising answer!</p>
<h2 id="worthless-effect-tracking">Worthless Effect Tracking</h2>
<p>Effect tracking, whether at the coarse or fine-grained level, is simply not commercially valuable. If it were, not only would there exist popular annotation processors for it, but the Java language itself would probably have the feature.</p>
<p>Instead, while the feature can be found as one of many features in the <a href="https://checkerframework.org/manual/#purity-checker">Checker Framework</a>, it has no commercial traction.</p>
<p>Moreover, early versions of PureScript used row types to provide statically-checked and fine-grained “effect tracking”, but the feature was <a href="https://github.com/purescript-deprecated/purescript-eff">quickly abandoned</a>, as it provided no commercial value, but rather slowed down feature development without benefit.</p>
<p>Contrast these results with lambdas (anonymous functions), which have made their way into every programming language because of the unquestionably positive effects they have on productivity.</p>
<p>It’s my contention that effect tracking is worthless precisely because if a developer has <em>any</em> idea about what a function is intended to do, then they already know with a high degree of certainty whether or not the function performs side-effects (assuming, of course, they are taught what it means for something to “perform side-effects”).</p>
<p>No developer who knows what side-effects are is surprised that <code class="language-plaintext highlighter-rouge">System.currentTimeMillis()</code> performs a side-effect, because they already know what the function is intended to do. Indeed, many language features and best practices (as well as IDE features!) are designed <em>precisely</em> to give developers a better idea of what functions are supposed to do.</p>
<p>Given a rudimentary understanding of what a function’s purpose, the purity of the function can be predicted with high probability (violations, like <code class="language-plaintext highlighter-rouge">java.net.URL#hashCode/equals</code>, become legendary!). However, given the purity of a function, this information by itself does not convey any useful information on the function’s purpose.</p>
<p>Stated simply, effect tracking isn’t incredibly useful in practice, because when it matters (and it doesn’t always matter), we already know roughly what functions do, and therefore, whether or not they perform side-effects.</p>
<p>Whatever benefit effect tracking would have would be overwhelmed by the cost of ceremony and boilerplate—this is doubly-true for the “fine-grained” variants.</p>
<p>Moreover, even assuming, evidence to the contrary notwithstanding, that effect tracking <em>was</em> a killer feature, it could be baked into an IDE without any modifications to a language’s syntax or semantics.</p>
<p>One could imagine clicking on an “interactions” button next to a function, and seeing what external (and side-effecting) systems each function interacts with. A pure tooling solution like this would have only upside, because it would not require mindless and verbose boilerplate and ceremony.</p>
<h2 id="why-io">Why IO?</h2>
<p>If effect tracking is commercially useless, then why use <code class="language-plaintext highlighter-rouge">IO</code> data types?</p>
<p>If you’re in Haskell, you don’t have a choice: if you want to get useful work done, then you will use <em>some</em> model of interaction with the outside world, and it may as well be <code class="language-plaintext highlighter-rouge">IO</code>, which is industry-proven and adopted extensively.</p>
<p>If you’re in Scala, however, you <em>do</em> have a choice: you can write plain vanilla Scala code, and perform side-effects anywhere that you want. Or you can grab one of the functional effect systems like <a href="https://zio.dev">ZIO</a>, and create and compose values that <em>model</em> side-effects instead.</p>
<p>The greatest reason to use a functional effect system like ZIO is that it makes side-effects <em>first-class values</em>. Values are things you can accept and return from functions. You can store them in data structures. You can write your own operators, which accept functional effects, and which transform or combine them in custom ways.</p>
<p>All of these abilities let you make your own control flow structures and factor out kinds of duplication you can’t avoid when you solve problems using side-effecting code (take a look at some of my talks for examples).</p>
<p>ZIO goes beyond providing first-class effects, and delivers additional valuable features, including:</p>
<ul>
<li>Highly-scalable fiber-based runtime</li>
<li>Resource-safety across asynchronous and concurrent effects</li>
<li>Declarative concurrency without locks and condition variables</li>
<li>Easy and safe parallelism</li>
<li>Context propagation and dependency injection</li>
<li>Execution traces that work across async and concurrent boundaries</li>
<li>Fiber-dumps that show the runtime graph of your application</li>
<li>Powerful operators that allow you to snap together solutions to complex problems quickly</li>
<li>Type-safety and user-friendliness</li>
<li>Features for making user-land code fully testable</li>
</ul>
<p>The real reason for using ZIO or other functional effect <strong>isn’t</strong> effect tracking: it’s everything else!</p>
<h2 id="summary">Summary</h2>
<p>Effect tracking isn’t a good reason to use an <code class="language-plaintext highlighter-rouge">IO</code> like data type, in Scala or any other programming language. That’s because effect tracking (which is actually a misnomer!) isn’t commercially useful.</p>
<p>If we have a vague idea about what functions do, then we generally have a really good idea about whether they perform side-effects, and compiler-enforced effect tracking would add overhead that wouldn’t pay for itself (assuming our only tangible benefit were “effect tracking”). Also, if we wanted effect tracking, we could always obtain the feature with non-invasive tooling, which could give us the same insight into function interactions without overhead.</p>
<p>Instead, functional effect systems like ZIO (and Haskell’s <code class="language-plaintext highlighter-rouge">IO</code> data type) let us take side-effects and make them more useful, by turning them into values, which we can transform and compose, solving complex problems with easy and type-safe combinators that simply can’t exist for side-effecting statements.</p>
<p>Beyond this ability, ZIO in particular gives us new superpowers, like easy and safe concurrency, parallelism, resource handling, dependency injection, diagnostic and debugging information, testability, and type-safety.</p>
<p>In summary, <em>don’t</em> use ZIO or IO or any functional effect system for effect tracking, because that benefit alone cannot pay for the cost.</p>
<p>Instead, if you decide to use functional effect systems, then use them to become a more powerful and productive programmer!</p>
<p><a href="https://degoes.net/articles/no-effect-tracking">Effect Tracking Is Commercially Worthless</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on May 03, 2020.</p>https://degoes.net/articles/zio-history2020-04-15T00:00:00-06:002020-04-15T00:00:00-06:00John A De Goeshttps://degoes.netjohn@degoes.net<p>When I <a href="https://degoes.net/articles/scalaz8-is-the-future">started</a> what would eventually become <a href="https://zio.dev">ZIO</a>, my goal was to give Scala a better asynchronous effect system, one that had some of the high-end features of Haskell’s IO monad, including cancellation of running effects and easy concurrency.</p>
<p>While developing this effect system, I was <a href="https://gist.github.com/jdegoes/97459c0045f373f4eaf126998d8f65dc">teaching</a> a lot of Scala developers functional programming professionally, covering topics like higher-kinded type classes (the functor hierarchy), monad transformers, free monads, and more. In addition, I was traveling the world speaking to Scala developers everywhere (including non-functional programmers), and listening to their feedback on functional programming, even and especially when hostile.</p>
<p>What I learned from my teaching experience and conversations with developers would radically change my perspective on the practicality of Haskell-style functional programming in Scala.</p>
<h2 id="real-world-feedback">Real World Feedback</h2>
<p>While live-coding for students, I realized how painful it was to use monad transformers; and how difficult it was to justify all this machinery to do simple things with state and errors.</p>
<p>Together with my students, I recoiled in horror so many times at just how much type inference you had to sacrifice for higher-kinded types.</p>
<p>I struggled to justify to skeptical everyday Scala programmers the tangible benefits that developers get from ‘effect-polymorphism’ beyond just testability.</p>
<p>I learned, both from my experience and the experience of other developers, just how difficult it is for someone who loves functional programming to sell the benefits to someone who does not love functional programming.</p>
<p>Through these experiences, I came to believe that if functional Scala continued to be <em>Haskell on the JVM</em>, it would never be particularly relevant for the broader market of programmers who just want to get stuff done.</p>
<p>These beliefs started to influence the design of ZIO.</p>
<h2 id="zio-evolves">ZIO Evolves</h2>
<blockquote>
<p>No plan of operations extends with any certainty beyond the first contact with the main hostile force
— Helmuth von Moltke the Elder, 1871</p>
</blockquote>
<p>Confronted with the often painful and unsatisfying reality of doing Haskell in Scala—a reality that drove many a functional programmer from Scala to Haskell (notably my friends Tony Morris, Sam Halliday, Emily Pillmore, and many others)—I decided to adapt:</p>
<ul>
<li>Having seen countless developers struggle with <code class="language-plaintext highlighter-rouge">EitherT</code>, I thought instead of fixing ZIO’s error type to be <code class="language-plaintext highlighter-rouge">Throwable</code>, I could make it polymorphic, and eliminate the poor ergonomics of <code class="language-plaintext highlighter-rouge">EitherT</code>. I was very happy with the result, because not only was it very pleasant to use, but you could see at compile-time which parts of your code can fail and which parts can’t, which I found to be a huge boost to productivity and correctness.</li>
<li>After a successful small-scale experiment, my friend and co-contributor <a href="https://twitter.com/wiemzin">Wiem Zine Elabidine</a> and I went through and aggressively refactored ZIO to use declaration-site variance everywhere. It was a massive refactoring, and at the time, it was extremely contentious, resulting in a lot of negative feedback. Scalaz and Cats had a philosophy of <em>variance is bad, always avoid it</em>. Yet after we completed the refactoring, it immeasurably increased the pleasure of using ZIO. For the first time, everything just inferred beautifully, without type annotations. Since then, I’ve even become obsessed with pushing variance to maximize type inference.</li>
<li>I started changing names of data types and operators, breaking with decades old Haskell traditions. For example, I renamed <code class="language-plaintext highlighter-rouge">point</code> / <code class="language-plaintext highlighter-rouge">pure</code> to <code class="language-plaintext highlighter-rouge">ZIO.succeed</code>, among many other changes. These changes resulted in a lot of negative feedback from functional programmers, culminating most recently in a lot of negative feedback about <code class="language-plaintext highlighter-rouge">ZIO.foreach</code>. My belief is not that jargon is useless (it’s essential among trained professionals), but that piggybacking on top of even imprecise domain knowledge can improve the onboarding experience for new developers.</li>
<li>I started changing documentation to emphasize practical pain points that developers face. I came to believe that no developer has a pain point called “lack of purity”. But they do have pain points around asynchronous programming; they have pain points around resource safety; they have pain points around concurrency; they have pain points around testability. ZIO’s current marketing focused emerged when the short pitch was changed from <em>a purely functional IO monad</em> to <em>a library for async & concurrent programming</em>.</li>
</ul>
<p>At the same time, ZIO-specific features like fibers and a pure (and automatic) cancellation model started influencing the design of Cats Effect, and I strongly argued for many changes in Cats Effect, most of which made the 1.0 release of the library (indeed, 1.0 was radically different because of these discussions!).</p>
<h2 id="divergence">Divergence</h2>
<p>Typed errors put some distance between the Cats Effect type class hierarchy and ZIO. If you wanted to reason about the error behavior of your programs statically, you could not use the Cats Effect type classes, because the type classes have no support for polymorphic errors.</p>
<p>In addition, a growing number of combinators on the ZIO effect type—combinators that did not exist in Cats Effect, Cats, or anywhere else—made it increasingly painful to restrict oneself to the subset of features supported by Cats Effect.</p>
<p>These growing seeds of divergence led to more direct use of ZIO among early adopters.</p>
<h2 id="zio-environment">ZIO Environment</h2>
<p>Early versions of ZIO only had typed errors. But in late 2018, after struggling in a class with the horrid contortions and complex machinery necessary to test functional programs using final-tagless, I had an idea to exploit contravariance to enable Scala to infer intersecting requirements in the composition of multiple effects.</p>
<p>I prototyped the solution in about a hundred lines of self-contained code, and in a fit of excitement, shared it with <a href="https://twitter.com/wiemzin">Wiem</a>, exclaiming, “This is gonna change things forever!”. Unfortunately, the prototype required a third type parameter; but on the flip side, it enabled a way of testing effectful programs that inferred flawlessly and didn’t require any type classes, implicits, higher-kinded types, or category theory.</p>
<p>Over the course of two months, <a href="https://twitter.com/wiemzin">Wiem</a> and I introduced the third “environment” type parameter into the ZIO effect type, in a massive refactoring that spanned thousands of lines of code. This moment was the birth of what we know now as <code class="language-plaintext highlighter-rouge">ZIO</code>: an effect type capable of propagating errors and successes upward; and propagating dependencies downward—the two primary needs of every functional program ever written!</p>
<p>I introduced ZIO environment to make testing functional programs easier, which I had come to believe was the only significant, tangible and sellable benefit of tagless-final. However, I quickly discovered that the <code class="language-plaintext highlighter-rouge">R</code> type parameter, combined with primitives already in ZIO (such as <code class="language-plaintext highlighter-rouge">Ref</code>), provides a way to model state and writer effects. Thus, there was no need for “MTL” anymore, as the principal transformers (<code class="language-plaintext highlighter-rouge">StateT</code>, <code class="language-plaintext highlighter-rouge">ReaderT</code>, <code class="language-plaintext highlighter-rouge">WriterT</code>, <code class="language-plaintext highlighter-rouge">ExceptT</code>) were already totally subsumed in the ZIO effect type.</p>
<p>Without explicitly trying, ZIO had become an alternative to MTL and even (optionally) tagless-final: only an alternative with excellent performance (something not possible in Scala with monad transformers), perfect type inference, and minimal knowledge prerequisites.</p>
<p>ZIO had introduced a different way to do functional programming in Scala. One with no jargon, no type classes, no implicits, no higher-kinded types, and no category theory.</p>
<h2 id="zio--cats-effect">ZIO & Cats Effect</h2>
<p>Despite these achievements, I did not view ZIO as a “competitor” to Cats Effect. I viewed Cats Effect as being primarily about interop between the different effect types through type classes, and I even argued that Cats IO should be a separate project to avoid conflict of interest.</p>
<p>I wrote about the <a href="https://degoes.net/articles/zio-cats-effect">synergy between the projects</a> and encouraged people to see ZIO as the <em>best implementation</em> for the Cats Effect type classes, because ZIO has improved semantics on Cats IO.</p>
<p>Some of the issues with Cats IO include the following:</p>
<ul>
<li>An <code class="language-plaintext highlighter-rouge">evalOn</code> implementation that does not work across async shifts and does not have proper error behavior;</li>
<li>A lack of fine-grained control over interruption;</li>
<li>The creation of “zombie” fibers that permanently hang any fiber that joins them, nullifying resource safety guarantees;</li>
<li>Async effects that resume on the caller’s thread pool;</li>
<li>Etc.</li>
</ul>
<p>Despite the synergy, I was quite concerned at the growing difference in expressive power between ZIO and the Cats Effect type classes.</p>
<p>On the one hand, ZIO defined an effect type with polymorphic (rather than monomorphic) errors, and with built-in polymorphic context propagation; together with improved semantics around interruption, error handling, shifting, and much more. But the Cats Effect type classes were rigidly locked to monofunctor effect types, and with increasingly dated semantics beset by confusing edge cases.</p>
<h2 id="life-after-cats-effect">Life After Cats Effect</h2>
<p>To narrow the growing gap between ZIO and the Cats Effect type classes, I volunteered to help design the next version of Cats Effect. Toward this end, I sketched out a complete hierarchy and <a href="https://github.com/typelevel/cats-effect/issues/321#issuecomment-449030858">submitted it for feedback</a>, conservatively aiming only to support typed errors, along with fixes for core semantic issues.</p>
<p>Ultimately, however, I was <a href="http://web.archive.org/web/20190905210318/https://gist.github.com/djspiewak/39fcf30fc4480abb5096010886558792">banned by Typelevel</a> in an elaborate and bizarre <a href="https://typelevel.org/blog/2019/09/05/jdg.html">press release</a> that was <a href="https://twitter.com/adamwarski/status/1169664500087304192">widely</a> <a href="https://twitter.com/propensive/status/1170098824242782208">denounced</a>—allegedly for ‘annoying’ technical debates about the ideal design of the library. Subsequently, my sketch was rejected, although many elements survived in the successor proposal for CE3.</p>
<p>The realization that Cats Effect would <strong>never</strong> catch up to ZIO, together with the massive pains of tagless-final, caused a tectonic shift in the market. Pretty soon, developers were building ZIO-specific libraries.</p>
<p>Many of the developers building these libraries first learned functional programming through ZIO. Mostly, they built these libraries without central coordination or planning or training. They just picked up ZIO by themselves and started building things they needed for production deployment.</p>
<p>Over time, it became clear to me that ZIO would have its own ecosystem, with a different focus and approach, and populated by a new and growing group of contributors from outside the historical functional Scala community.</p>
<h2 id="zio-today">ZIO Today</h2>
<p>Today, ZIO is first and foremost a library to help developers build <em>modern applications</em>.</p>
<p>Modern applications are asynchronous and concurrent, and they shouldn’t leak resources, they shouldn’t deadlock, they shouldn’t have race conditions, they should be easily testable, and they should use the Scala type system to catch bugs before they happen.</p>
<p>ZIO helps you quickly build very powerful and correct applications by playing to the strengths of Scala.</p>
<p>Yeah, ZIO happens to be purely functional. Yeah, it’s an alternative to MTL. And yeah, even though it still plays well with Cats Effect libraries, it’s increasingly its own thing with its own community and ecosystem.</p>
<p>But the big picture is about leveraging both functional programming and all the features of Scala (even the parts historically eschewed by functional programmers!) to make Scala developers really happy and really productive.</p>
<p>If we do that, the rest is just details.</p>
<p><a href="https://degoes.net/articles/zio-history">A Brief History of ZIO</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on April 15, 2020.</p>https://degoes.net/articles/coronapocalypse2020-04-03T00:00:00-06:002020-04-03T00:00:00-06:00John A De Goeshttps://degoes.netjohn@degoes.net<p>The world has changed unimaginably in the past few weeks.</p>
<p>Watching a show on Netflix last night, I was struck by how everyday events now seem so completely alien: traveling between countries, eating out at restaurants, using subways without face masks.</p>
<p>That was the world of a few weeks ago. A world that, with each passing day, becomes an ever-receding memory.</p>
<p>Without question, the COVID-19 pandemic has radically altered the entire world at a speed no generation has ever witnessed—because the world is more interconnected now than at any time in the history of our species.</p>
<p>This event will affect everyone on the planet. Including every software developer.</p>
<p>In addition to heavy costs on human life and health, the pandemic will derail careers and cost many tech jobs, as the largest recession in decades begins taking form.</p>
<p>In this post, I offer my tips to help developers prepare for what’s coming.</p>
<h2 id="the-economic-meltdown">The Economic Meltdown</h2>
<p>The economic impact of the pandemic started the instant people altered their spending habits. It intensified greatly as countries shut down flights, established curfews, enforced lockdowns, and ordered non-essential retailers to close shop.</p>
<p>Unlike tech giants like Apple and Google, small-time retailers don’t have the luxury of hoarding profit that they can use to weather economic storms.</p>
<p>Instead, retailers have significant liabilities, ranging from lease agreements, to employees, to purchase orders, to business loans. Due to competition, they frequently operate at close to margins, which means they are vulnerable to being crushed by negative swings of any significant magnitude or lasting duration.</p>
<p>Their options for reacting to economic downturns are limited:</p>
<ul>
<li>Cost cutting measures (principally layoffs)</li>
<li>Delaying payments to suppliers</li>
<li>Increasing debt</li>
</ul>
<p>Relying too much on any of these tactics results in a death spiral; but worse, with the growing recession, the latter two tactics are becoming less viable.</p>
<p>Retailers are going to be hit hard, and many of them will <em>never</em> recover.</p>
<p>National and international chains selling non-essential goods will be forced to cut costs as revenue plummets, which means some tech workers will lose their jobs.</p>
<p>However, most smaller retailers don’t employ developers. Does that mean developers will be insulated from the recession?</p>
<p>Not even remotely! The entire economy is interconnected. What happens to some of us affects all of us.</p>
<p>Retailers leverage B2B software to run their business (inventory management, supply chain management, point-of-sale software, marketing software, and much more), because they don’t have technical resources themselves. Retailers also spend a lot of money on advertising, which sustains massive portions of the freemium tech economy—ranging from search engines to email providers to mobile games to social networks.</p>
<p>Retailers pay their employees, who spend some portion of their income on consumer web apps, consumer mobile apps, and various other forms of consumption services powered by the tech industry. As these employees lose their jobs, their disposable income drops, which negatively impacts some consumer tech companies.</p>
<p>Moreover, as the pandemic has raged on, the impact has gone way beyond just retailers. The travel, services, and hospitality industries have already nose-dived—without hope of a near-term recovery—and these industries count in their number not just tech companies like Airbnb, Yelp, Expedia, and many others, but also innumerable software vendors that facilitate logistics, supply chain management, resource planning, and much more.</p>
<p>The disruption of retail, travel, hospitality, and services will all impact pure-play tech companies, both directly in the ways described above, and also indirectly, as second-level effects ripple throughout the interconnected economy.</p>
<p>The full economic impact of the pandemic is hard to predict, but it’s manifestly clear that these changes will lead to the loss of many developer jobs.</p>
<h2 id="a-survival-strategy">A Survival Strategy</h2>
<p>As others have discussed, one can minimize the impact of a recession by maximizing one’s <em>personal runway</em>, which involves cost-cutting measures (including, potentially, re-location to areas with a lower cost of living).</p>
<p>Beyond economic preparation, however, I encourage developers to take an active role in their career development, using one or more of the following tactics:</p>
<ol>
<li><strong>Orient</strong>. Develop a model of the effects of the pandemic on hiring and layoffs in different industries.</li>
<li><strong>Focus</strong>. Don’t let distraction become a liability during the crisis, but rather, turn focus into an asset.</li>
<li><strong>Adapt</strong>. Modify your unique value proposition as a software engineer so that you become more valuable to your employer.</li>
<li><strong>Improve</strong>. Improve in-demand knowledge and skills in an effort to differentiate yourself from others.</li>
<li><strong>Innovate</strong>. If you are so inclined, use your skills as a software developer to help the world through this time.</li>
<li><strong>Don’t Panic</strong>. Life is full of setbacks, but where there is life, there is hope.</li>
</ol>
<p>I’ll expand on these recommendations in the sections that follow.</p>
<h2 id="orient">Orient</h2>
<p>The pandemic will affect some industries more than others, and your own career risk partially depends on what type of company you work for and what kind of project you work on.</p>
<p>Some likely candidates for early downsizing:</p>
<ul>
<li>Products targeting non-essential box-store retail, restaurant, travel, and hospitality industries;</li>
<li>Startups without exceptional metrics;</li>
<li>Consulting, outsourcing, and development shops that target primarily non-tech companies (excluding the government);</li>
<li>Expansion and speculative products and projects inside every industry; if there aren’t existing customers driving significant revenue, or internal users driving significant value, it’s a candidate for cancellation.</li>
</ul>
<p>Some industries will actually benefit from the pandemic, and could enjoy accelerated hiring, even while companies in other industries are forced to downsize.</p>
<p>The following types of companies should benefit:</p>
<ul>
<li>Companies enabling individuals and companies to shift more day-to-day activities to the cloud. For example, companies that facilitate remote collaboration, online retail, fulfillment, delivery, socializing, and so forth.</li>
<li>Low-cost entertainment, such as games and streaming films and shows.</li>
<li>Low-cost retailers of essential goods, such as groceries, medicine, household consumables.</li>
<li>Personal health and fitness materials and supplies, and survival goods.</li>
</ul>
<p>Though some companies will be downsizing, others will be hiring more developers, providing a source of new jobs to those who have the right knowledge, skills, and experience.</p>
<h2 id="focus">Focus</h2>
<p>The pandemic has thrust a justly distracted public into remote work environments that they are not properly equipped to deal with, resulting in lower productivity.</p>
<p>Further affecting productivity is continuous media coverage of the pandemic, including real-time reports on infection rates, early research into treatments and vaccine developments, celebrity hospitalizations and deaths, and endless memes.</p>
<p>It’s easy to spend hours a day keeping up with what’s going on. While some information is essential, because it can lead to actions that help you improve or save lives, consuming too much news can impact your work productivity.</p>
<p>In hard times, managers are sometimes asked to rank employees in order of least essential to most essential. These “kill lists” make it easier to cut costs quickly and precisely in response to slashed budgets and cancelled projects.</p>
<p>When your company does layoffs—and for many companies, it will be <em>when</em>, not <em>if</em>—you probably don’t want to be at the top of the kill list, especially in a recession.</p>
<p>So stay as informed as you need to be, but also stay focused on your career.</p>
<h2 id="adapt">Adapt</h2>
<p>Hard times change people and they change companies—everyone becomes more risk averse, less open, and more fiscally conservative.</p>
<p>While companies <em>always</em> care about the bottom line, a recession of unknown duration and magnitude means there is intense pressure to <em>do more with fewer resources</em>.</p>
<p>Developers who deeply internalize this imperative and adapt will have more successful careers than those that don’t.</p>
<p>Here are some tangible ways you can adapt to uncertain times:</p>
<ul>
<li><strong>Minimize reinvention</strong>. Many developers love to build rather than reuse, partially because building results in a better solution, and partially because reusing existing solutions is painful due to learning curves and edge cases. Yet, while a booming economy funded extreme reinvention (some companies invented databases, programming languages, query languages, and far more!), a recession demands a frugal use of developer time, and one way to do that is to reuse as much software as possible. <em>Become good at figuring out how to reuse and adapt existing software for new use cases.</em></li>
<li><strong>Maximize productivity</strong>. Frameworks that are closer to a business problem can allow higher productivity than a random assortment of libraries; cutting the right corners can accelerate time-to-feedback; proven approaches can be less risky than experimental approaches; static type systems and property checking can reduce the need for voluminous hand-written automated tests. <em>Draw from your strengths to help you and your team deliver more value to the business in less time.</em></li>
<li><strong>Minimize expenses</strong>. Cloud infrastructure and costly web services can add up to hundreds of thousands or millions, even for relatively small companies. There will be increasing pressure to minimize these costs, and to replace high-priced services with those that have low total cost of ownership. <em>Be proactive about these cost-cutting measures, looking for ways to save your company money.</em></li>
</ul>
<h2 id="improve">Improve</h2>
<p>Even in the best of times, the tech industry changes rapidly. Languages, frameworks, libraries, and technologies all rise and fall.</p>
<p>Beyond constant technological change, the relevancy of every developer’s knowledge and skills changes continuously. Some knowledge and skills that used to be in-demand (such as debugging ActiveX components running in Internet Explorer) become irrelevant.</p>
<p>Recessions are hard on everyone, but they are especially hard on developers who stagnate.</p>
<p>In times like these, push yourself to learn more. Shore up your weak skills. And even if you’re lucky enough to have a great employer, don’t count on your employer to invest in your education, because educational budgets won’t last long in a recession.</p>
<h2 id="innovate">Innovate</h2>
<p>Necessity, as they say, is the mother of invention, and social and economic challenges open new doors to innovation.</p>
<p>For the very entrepreneurial, as society attempts to survive the pandemic, and then rebuild around new values and modes of interaction, there will be countless opportunities for innovation at every level.</p>
<p>In the early days, this includes helping individuals be successful working and living in the cloud, and dealing with the devastation wrought by growing unemployment and the recession; and helping smaller businesses transition into a cloud-first mindset.</p>
<p>Software developers are makers, and as makers, we have the ability to create software that is part of the solution to these problems.</p>
<p>Beyond just opportunities for product-level innovation, there will be opportunities in the world of open source. There will be demand for new kinds of technologies, and more productive solutions to pervasive and resource-consuming pains.</p>
<h2 id="dont-panic">Don’t Panic</h2>
<p>No matter how much any of us prepares, there are no guarantees in life—other than death and taxes.</p>
<p>If you do lose your job, try not to panic: a lot of changes are coming in the world, but for the foreseeable future, the need for skilled software developers remains constant.</p>
<p>If you pay attention to demand, and are willing and able to adjust your knowledge and skills in response to the changing environment, you <em>will</em> find a job.</p>
<h2 id="summary">Summary</h2>
<p>The pandemic has changed the world forever. These changes are rapidly spreading throughout the interconnected global economy, and while early ripples of a growing recession can be felt now, the worst is still ahead.</p>
<p>Many startups will not survive, and many developers across even some large tech companies will be laid off—not all at once, but in waves, as financial stress propagates from sector to sector, industry to industry, company to company.</p>
<p>Aside from being frugal, developers can prepare for these changes by being aware of the global economic picture; by staying focused; by adapting to times of scarcity; by improving their knowledge and skills; and, if so inclined, by innovating in areas that they know well.</p>
<p>Above all, panicking won’t help anyone. Stay calm. It’s going to be painful, and it’s going to take a while, but eventually the economy <em>will</em> recover.</p>
<p><a href="https://degoes.net/articles/coronapocalypse">A Developer's Guide to Surviving the Coronapocalypse</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on April 03, 2020.</p>https://degoes.net/articles/fp-glossary2019-12-22T00:00:00-07:002019-12-22T00:00:00-07:00John A De Goeshttps://degoes.netjohn@degoes.net<p>I’ve taught functional programming for years now, each time experimenting with different ways of teaching core concepts. Over time, I’ve collected and converged on simple (but reasonably precise) pedagogical definitions for a range of functional concepts.</p>
<p>In this post, I’ll share those definitions with you, in my first ever, <em>Glossary of Functional Programming</em>. Enjoy!</p>
<h2 id="glossary">Glossary</h2>
<p><strong>Abstraction</strong>. An <em>abstraction</em> is a precise description of the ways in which different data types share common structure. Abstraction allows different data types to participate in <em>generic programming</em>. In functional programming, abstractions are defined by <em>algebraic structure</em>, and are usually encoded in a programming language using <em>type classes</em>. Common examples of abstractions include monoids, functors, and monads.</p>
<p><strong>Ad Hoc Polymorphism</strong>. <em>Ad hoc polymorphism</em> is any language feature that allows overloading functions and operators (providing multiple implementations for the same symbol) in such a fashion that the compiler or runtime can automatically select which implementation to call based on the types involved in the function application. Examples of ad hoc polymorphism include method overloading and <em>type classes</em>.</p>
<p><strong>Algebra</strong>. An <em>algebra</em> is a set of <em>elements</em> together with a set of <em>operations</em> on the elements. The operations of an algebra are defined by <em>algebraic laws</em>, which give the operations meaning by relating them to each other. For example, addition on integers forms a very simple algebra, where the elements are integers, and the sole operation is addition, which satisfies assocative and commutative laws.</p>
<p><strong>Algebraic Data Type (ADT)</strong>. An <em>algebraic data type</em> is any data type composed from <em>product types</em> (records) and <em>sum types</em> (enumerations). In functional programming, algebraic data types are used for <em>data modeling</em>.</p>
<p><strong>Algebraic Law</strong>. An <em>algebraic law</em> is a <em>universally quantified</em> statement that asserts a relationship between the <em>operations</em> of an <em>algebra</em>. For example, an algebraic law for addition could assert that <code class="language-plaintext highlighter-rouge">(a + b) + c</code> is equal to <code class="language-plaintext highlighter-rouge">a + (b + c)</code> (<em>associativity</em>). In mainstream programming languages, algebraic laws are usually tested using <em>property-based testing</em>.</p>
<p><strong>Algebraic Structure</strong>. <em>Algebraic structure</em> is any <em>structure</em> that is defined using an <em>algebra</em>.</p>
<p><strong>Applicative</strong>. An <code class="language-plaintext highlighter-rouge">Applicative</code> Functor is an <code class="language-plaintext highlighter-rouge">Apply</code> Functor that allows taking a value and converting it into a <em>functional effect</em> that succeeds with the specified value. Under the view of functors as DSLs, <code class="language-plaintext highlighter-rouge">Applicative</code> adds a “pure return” statement to the DSL.</p>
<p><strong>Apply</strong>. An <code class="language-plaintext highlighter-rouge">Apply</code> Functor is a Functor that allows zipping two <em>functional effects</em> together, to get a tuple of their successes. Under the view of functors as DSLs, <code class="language-plaintext highlighter-rouge">Apply</code> adds a way to combine two programs together into one program, and preserve both successes.</p>
<p><strong>Associativity</strong>. An <em>associative</em> operator is a binary operator that, when applied to three values, always results in the same value, regardless of the order in which pairwise values are combined. For example, addition on numbers is associative, because for all numbers <code class="language-plaintext highlighter-rouge">a</code>, <code class="language-plaintext highlighter-rouge">b</code>, and <code class="language-plaintext highlighter-rouge">c</code>, <code class="language-plaintext highlighter-rouge">(a + b) + c</code> is equal to <code class="language-plaintext highlighter-rouge">a + (b + c)</code>.</p>
<p><strong>Category Theory</strong>. <em>Category theory</em> is a branch of mathematics that defines mathematical structure using directed labeled graphs (<em>categories</em>). An alternative to algebraically-defined structure, category theory is a powerful and precise pattern language for mathematics, computer science, and functional programming. Knowledge of category theory is helpful but not necessary for mastery of applied functional programming.</p>
<p><strong>Commutativity</strong>. A <em>commutative</em> operator is a binary operator that produces the same result no matter the order of the operands. For example, addition is commutative because <code class="language-plaintext highlighter-rouge">a + b</code> is equal to <code class="language-plaintext highlighter-rouge">b + a</code>, for all numbers <code class="language-plaintext highlighter-rouge">a</code> and <code class="language-plaintext highlighter-rouge">b</code>.</p>
<p><strong>Composable</strong>. An operator is <em>composable</em> if it can be applied to its own return value. For example, the addition operator is composable with respect to integers, because it returns another integer, which can then be added to other integers. Similarly, the boolean negation operator is composable, because it is possible to negate a boolean value that has itself been negated. Composability allows a small number of operators (which can be reasoned about simply) to have immense expressive power.</p>
<p><strong>Contravariant Functor</strong>. A <code class="language-plaintext highlighter-rouge">Contravariant</code> Functor is a type class that allows transforming the input type to some <em>functional effect</em> using a function. Contravariant functors are often formed by polymorphic types that accept input (such as functions, or stages in a pipeline).</p>
<p><strong>Data Modeling</strong>. <em>Data modeling</em> is the act of constructing a data model of domain objects (such as people, products, schedules, etc.). A goal of data modeling in statically-typed functional programming is to construct a data model so precise, it is impossible to construct bad data, a process sometimes called, <em>making illegal states unrepresentable</em>.</p>
<p><strong>Declarative Programming</strong>. <em>Declarative programming</em> is a style of programming in which solutions are constructed by specifying goals, rather than specifying the sequential steps necessary to achieve these goals (<em>what</em> instead of <em>how</em>). Declarative programming is the opposite of imperative programming. Some languages, such as SQL and Datalog, are inherently declarative; but declarative programming can be practiced in any programming language, typically by defining declarative DSLs atop imperative implementations.</p>
<p><strong>Dependency Injection</strong>. <em>Dependency injection</em> refers to a framework, library, language feature, or architectural pattern that facilitates threading dependencies throughout an application, and wiring up those dependencies for different scenarios. Dependency injection is one of the primary architectural needs of large-scale, complex applications.</p>
<p><strong>Dependent Typing</strong>. <em>Dependent typing</em> is a method of defining types that permits types to depend on values. For example, in dependently typed programming languages, one can define the type of vectors whose length is equal to some value. Examples of dependently typed programming languages include Idris, Coq, and Agda. Dependently typed programming languages allow proving more properties about programs at compile-time, albeit at increased development cost.</p>
<p><strong>Deterministic</strong>. A <em>deterministic</em> procedure returns the same output for the same input.</p>
<p><strong>Domain-Specific Language (DSL)</strong>. A <em>domain-specific language</em> (DSL) is a “mini-language” that is designed to solve a certain subproblem in an application. In functional programming, DSLs are generally <em>embedded</em>, which means users of these DSL use data types and operators, defined in the host language, to construct programs in the DSL.</p>
<p><strong>Effect Rotation</strong>. <em>Effect rotation</em> refers to the construction of highly polymorphic data types (typically <em>indexed monads</em>) that simultaneously support multiple <em>functional effects</em>. Each effect supported by the data type is represented with a different type parameter, and each effect may be introduced or eliminated using certain operations. Effect rotation provides much of the benefit of monad transformers, but with substantially improved performance, and in some languages like Scala, significantly better type inference.</p>
<p><strong>Existential Quantification</strong>. <em>Existential quantification</em> asserts that a statement holds for some choice of a given variable. For example, the value <code class="language-plaintext highlighter-rouge">list: List[_]</code> is existentially quantified over the <code class="language-plaintext highlighter-rouge">List</code> type parameter, asserting that there exists some (unknown) type that describes the type of elements in the list, without specifying what this type is.</p>
<p><strong>Existential Type</strong>. An <em>existential type</em> is a specific, definite type, which is unknown to code interacting with the type. In programming languages, existential types are used to “forget” information that need not be carried around in type parameters. In Scala, abstract type members are always existential types.</p>
<p><strong>Expression</strong>. An <em>expression</em> is a value constructed from the application of functions or operators to other values. The parameters to the functions or operators are called the <em>terms</em> of the expression. For example, <code class="language-plaintext highlighter-rouge">a + b</code> is an expression, which computes the result of adding the term <code class="language-plaintext highlighter-rouge">a</code> to the term <code class="language-plaintext highlighter-rouge">b</code>.</p>
<p><strong>F-Algebra</strong>. An F-algebra is a generalization of algebraic structure that comes from category theory, which makes it possible to represent algebraic laws without universal quantification, using only morphisms. In functional programming, F-algebras appear more directly as interpreters for both tagless-final and free monadic programs (<em>natural transformations</em>), and as algebras and coalgebras in recursion schemes.</p>
<p><strong>First-Class</strong>. <em>First-class</em> constructs are those which can be created, manipulated, and abstracted over using the most powerful machinery a language exposes for these tasks. For most programming language, <em>first-class</em> means the construct is a <em>value</em>. First-class constructs give programmers much more power than second-class constructs. Some constructs which are not first-class can nonetheless be <em>reified</em> to obtain some of the benefits of being first-class.</p>
<p><strong>Flow analysis</strong>. Type-directed <em>flow analysis</em> is a process whereby the flow of information in the declaration of a data type or a function is analyzed using type information alone. Performing flow analysis can yield useful information about the implementation of polymorphic declarations. For example, for a function <code class="language-plaintext highlighter-rouge">def foo[A](a: A): A</code> (<code class="language-plaintext highlighter-rouge">foo :: a -> a</code>), one can tell using flow analysis that the output of the function must have come from its single input parameter, because there is no other way for the function to return a value of the polymorphic type.</p>
<p><strong>Free Structure</strong>. A <em>free structure</em> is a data structure that <em>reifies</em> operations of some algebra into a tree-like data structure. Later, the data structure can be traversed, and the operations applied to concrete values—a process called <em>interpretation</em> of the free structure. An example free structure is <code class="language-plaintext highlighter-rouge">List</code> (<code class="language-plaintext highlighter-rouge">[]</code>), which is a free monoid for any type <code class="language-plaintext highlighter-rouge">A</code>; that is, for any type <code class="language-plaintext highlighter-rouge">A</code>, <code class="language-plaintext highlighter-rouge">List[A]</code> (<code class="language-plaintext highlighter-rouge">[a]</code>) provides both a neutral value (the empty list) and an append operation (list concatenation). Another example is <code class="language-plaintext highlighter-rouge">Free</code>, which provides a free monad for any type constructor <code class="language-plaintext highlighter-rouge">F[_]</code> (<code class="language-plaintext highlighter-rouge">f : * -> *</code>).</p>
<p><strong>Function</strong>. A mathematical <em>function</em> <code class="language-plaintext highlighter-rouge">f : A => B</code> (<code class="language-plaintext highlighter-rouge">f :: a -> b</code>) associates every value in a set <code class="language-plaintext highlighter-rouge">A</code> (called the <em>domain</em>) with a single value in a set <code class="language-plaintext highlighter-rouge">B</code> (called the <em>codomain</em>). For a given function, this mapping is static (it does not change). In programming languages, the domains and codomains of value-level functions are types, and all such functions are <em>total</em> (they map every input to some output), <em>deterministic</em> (they map the same input to the same output), and <em>pure</em> (they only map inputs to outputs).</p>
<p><strong>Functional Effect</strong>. A <em>functional effect</em> is an immutable data type that describes (or <em>models</em>) the computation of one or more values, where the computation may require an additional feature like optionality, logging, access to context (like configuration), errors, state, or input/output. Using effect-specific operations, functional effects can be transformed and composed to model solutions to complex problems out of solutions to smaller problems. The ways of constructing a functional effect, together with the operations on effects of that type, form a DSL, whose capabilities are dictated by the constructors and operations. Frequently, functional effects are <em>run</em> or <em>interpreted</em> into plain values or into other functional effects. Common functional effects include <code class="language-plaintext highlighter-rouge">Option</code> (<code class="language-plaintext highlighter-rouge">Maybe</code>), which models missing values; <code class="language-plaintext highlighter-rouge">Either</code>, which models failure; and <code class="language-plaintext highlighter-rouge">ZIO</code> (<code class="language-plaintext highlighter-rouge">IO</code>), which models side-effects, including asynchronous side-effects.</p>
<p><strong>Functional Programming</strong>. <em>Functional programming</em> is a style of programming in which solutions are constructed by defining and applying (mathematical) <em>functions</em>. Many programs are not “purely” functional, but contain parts that are written in a functional style, as well as parts that are written in a procedural style. Functional programming can be practiced in statically-typed as well as untyped programming languages.</p>
<p><strong>Functional-Imperative</strong>. <em>Functional-imperative</em> is a hybrid style of programming that uses higher-order functions to embed an imperative model of computation inside functional code. Functional-imperative style relies on <em>monads</em> or an equally powerful abstraction to represent the stateful sequentiality of imperative computations.</p>
<p><strong>Functor</strong>. A <code class="language-plaintext highlighter-rouge">Functor</code> is a type class that allows <em>mapping</em> the success value of some <em>functional effect</em> using a function. Every functor can be regarded as a type of DSL, where the terms in the functor sum type correspond to different instructions in the DSL. Under this view of functors as DSLs, <code class="language-plaintext highlighter-rouge">Functor</code> provides a way to change the success value of a DSL “program” into some other success value, by applying a specified function.</p>
<p><strong>Generalized Algebraic Data Type (GADTs)</strong>. <em>Generalized algebraic data types</em> are polymorphic ADTs whose component types may introduce existential types or impose type equality constraints on specific type parameters. GADTs enable easier modeling and use of type-safe DSLs.</p>
<p><strong>Generic Programming</strong>. <em>Generic programming</em> is a style of programming that uses <em>polymorphism</em> to “forget” irrelevant details about data types, which allows code to be used across many different data types that share common structure. Generic programming can be thought of as defining a “template” that is then specialized to concrete types.</p>
<p><strong>Global Reasoning</strong>. <em>Global reasoning</em> is a property of some code wherein the correctness of the code cannot be inferred without considering prior application state or all possible inputs.</p>
<p><strong>Higher-Kinded Types</strong>. <em>Higher-kinded types</em>, more precisely called <em>higher-kinded generics</em>, is a language feature in which type parameters may have a kind higher than <code class="language-plaintext highlighter-rouge">*</code>. The two most mainstream languages with higher-kinded types are <em>Scala</em> and <em>Haskell</em>.</p>
<p><strong>Higher-Order</strong>. <em>Higher-order</em> refers to higher-order functions, which are functions that accept functions as input, or return functions as output. For example, <code class="language-plaintext highlighter-rouge">map</code> on a list is a higher-order function, because it accepts a function as one of its parameters; similarly, the <code class="language-plaintext highlighter-rouge">Monad</code> type class is a higher-order type class, because the kind of its type parameter is <code class="language-plaintext highlighter-rouge">(* => *) => *</code>, which is a higher-order type-level function.</p>
<p><strong>Homomorphism</strong>. <em>Homomorphisms</em> are a special type of function <code class="language-plaintext highlighter-rouge">f : A => B</code> that preserve a given <em>algebraic structure</em>. For example, the square function (which returns its input multiplied by itself) is a semigroup homomorphism, because it preserves, for example, the multiplicative semigroup.</p>
<p><strong>Imperative Programming</strong>. <em>Imperative programming</em> is a style of programming in which solutions are constructed from step-by-step instructions, where later instructions can depend on the result of previous instructions. In procedural programming, imperative programming is typically done with side-effecting statements, whereas in functional programming, imperative programming is typically done with <em>monads</em>.</p>
<p><strong>Indexed Monad</strong>. An <em>indexed monad</em> is a generalization of <code class="language-plaintext highlighter-rouge">Monad</code> for highly polymorphic data types having multiple type parameters, which form ordinary <code class="language-plaintext highlighter-rouge">Monad</code> instances for any possible choice of the extra type parameters. Indexed monads allow increased type precision, and allow operations that cannot be expressed with non-indexed monads. Examples of indexed monads included indexed state and <code class="language-plaintext highlighter-rouge">RIO</code> (in Haskell), and <code class="language-plaintext highlighter-rouge">ZIO</code> (in Scala).</p>
<p><strong>Indirection</strong>. <em>Indirection</em> is any programming language feature (interfaces, type classes, records of functions, module definitions) that allows multiple implementations of some required functionality to be defined and provided to code that requires these capabilities. Indirection facilitates testability, code reuse, and modularity, both in functional programming, and in other paradigms.</p>
<p><strong>Isomorphism</strong>. An <em>isomorphism</em> is an equivalence between two data types that have the same information. An isomorphism is defined by two functions: one to translate from the first data type to the second; and one to translate from the second data type to the first. These functions must satisfy “roundtrip” laws in order for them to form an isomorphism, which demonstrate the two representations contain the same information. In functional programming, many things are considered equal <em>up to isomorphism</em>. For example, <code class="language-plaintext highlighter-rouge">(A, Unit)</code> and <code class="language-plaintext highlighter-rouge">A</code> are <em>equal up to isomorphism</em>, meaning these two different types have the same information.</p>
<p><strong>Kind</strong>. A <em>kind</em> describes the structure of a type or type constructor. Monomorphic types like <code class="language-plaintext highlighter-rouge">Float</code> and <code class="language-plaintext highlighter-rouge">Boolean</code>, which take no type parameters, belong to the set of all types, which is denoted <code class="language-plaintext highlighter-rouge">*</code> (“star”). Type constructors like <code class="language-plaintext highlighter-rouge">List</code> (<code class="language-plaintext highlighter-rouge">[]</code>) and <code class="language-plaintext highlighter-rouge">Option</code> (<code class="language-plaintext highlighter-rouge">Maybe</code>) belong to the set of all type constructors that take one type and return one type, which is denoted <code class="language-plaintext highlighter-rouge">* => *</code> (“star to star”). All kinds higher than <code class="language-plaintext highlighter-rouge">*</code> are type-level functions, which accept types (or type constructors) and return types. For example, if you give the <code class="language-plaintext highlighter-rouge">List</code> type constructor the type <code class="language-plaintext highlighter-rouge">Boolean</code>, then you get back the type representing lists of boolean values. The terminology and notation of kinds allow us to talk about the ways in which different types and type constructors are similar or distinct.</p>
<p><strong>Lambda</strong>. A <em>lambda</em> is an anonymous function, which is a function value that has no name. Because lambdas are values, lambdas may be passed to functions, and returned from functions. All languages that support functional programming provide a way to encode lambdas.</p>
<p><strong>Lambda Calculus</strong>. The <em>lambda calculus</em> is a way of expressing arbitrary computation using only <em>lambdas</em>. The lambda calculus is an alternative to Turing machines and other calculi of computation, and gives meaning to the term <em>functional programming</em>. There are many different formulations of the lambda calculus, not just one.</p>
<p><strong>Lenses</strong>. <em>Lenses</em> are <em>optics</em> that allow getting and setting terms in <em>product types</em>. For example, a lens could allow getting and setting the <em>name</em> field inside objects of type <code class="language-plaintext highlighter-rouge">Person</code>. In the functional context, <em>setting</em> a term in a product means creating a new modified product, given the new value for the term, and the old product value.</p>
<p><strong>Liskov Substitution Principle</strong>. The <em>Liskov substitution principle</em> says that if a function accepts an input of type <code class="language-plaintext highlighter-rouge">A</code>, then it should be valid to pass the function any subtype of <code class="language-plaintext highlighter-rouge">A</code>, without altering the correctness of the function. Similarly, for a function returning an output of type <code class="language-plaintext highlighter-rouge">B</code>, it should be valid for the function to return a subtype of <code class="language-plaintext highlighter-rouge">B</code>. The Liskov substitution principle enables principled reasoning about subtyping.</p>
<p><strong>Local Reasoning</strong>. <em>Local reasoning</em> is a property of some code wherein the correctness of the code can be inferred locally under specified assumptions, without considering prior application state or all possible inputs.</p>
<p><strong>Map</strong>. To <em>map</em> over a polymorphic data type is to change one type parameter into another type, by specifying a way to transform values of that type. For example, a list of integers can be mapped into a list of strings, by specifying a way to transform an integer into a string. To be well-defined, and to constitute a valid <code class="language-plaintext highlighter-rouge">Functor</code>, mapping over any polymorphic data type with an identity function must yield something that is equal to the original value.</p>
<p><strong>Minimal</strong>. A set of operators is <em>minimal</em> if there exist no smaller set of orthogonal operators that has the same expressive power.</p>
<p><strong>Modular</strong>. A factoring of a solution to a problem is <em>modular</em> when it has been divided into independent concerns called <em>modules</em>, such that each module knows no more than necessary about other modules. Modularity helps tame the complexity of large-scale software development.</p>
<p><strong>Monad</strong>. A <code class="language-plaintext highlighter-rouge">Monad</code> is an <code class="language-plaintext highlighter-rouge">Applicative</code> Functor that allows combining a first <em>functional effect</em>, together with a function that takes the success value of the first effect and returns a second effect (<em>continuation</em>). Monads represent the essence of imperative programming: do a first thing, and then do this second thing, which depends on the value computed by the first thing. Under the view of functors as DSLs, <code class="language-plaintext highlighter-rouge">Monad</code> adds sequential “statements”, where subsequent statements depend on previous ones.</p>
<p><strong>Monad Transformer</strong>. A <em>monad transformer</em> is a data type that adds one functional effect atop any other functional effect. For example, the <code class="language-plaintext highlighter-rouge">OptionT</code> (<code class="language-plaintext highlighter-rouge">MaybeT</code>) monad transformer adds the effect of optionality to any other functional effect. A monad transformer, when “stacked” atop a monad, itself forms a monad, which allows building up large stacks of monad transformers, each layer adding some desired functional effect. Monad transformers have significant performance overhead in languages like Scala. An alternative to monad transformers is <em>effect rotation</em>.</p>
<p><strong>Monoid</strong>. A monoid is a semigroup equipped with a <em>neutral element</em> (often called <em>zero</em>), such that appending the neutral element to any other element (on either side) returns that same element unchanged.</p>
<p><strong>Monomorphic</strong>. <em>Monomorphic</em> is the opposite of <em>polymorphic</em>, and refers to a function that takes no type parameters (instead, all of its inputs are concrete types), or a data type that takes no type parameters (rather, it defined entirely with concrete types).</p>
<p><strong>Natural Transformation</strong>. A <em>natural transformation</em> is any polymorphic function between <em>Functors</em>.</p>
<p><strong>Nominal Typing</strong>. <em>Nominal typing</em> is a method of defining types solely based on their names. Two nominal types are equal if and only if they have the same fully-qualified name. Nominal typing stands in contrast to <em>structural typing</em>, in which two types that have different names but the same structure are regarded as the same.</p>
<p><strong>Onion Architecture</strong>. <em>Onion architecture</em> is a method of architecting programs that involves gradually translating business logic into lower-layer levels, until at the edges of the application, logic is translated into system calls. Applications written in the onion architecture bear similarities with multi-staged compilers, where every layer of the onion contains DSLs that are “compiled” into other DSLs.</p>
<p><strong>Optics</strong>. <em>Optics</em> are <em>values</em> that allow getting and setting smaller parts of a larger data structure, in a purely functional way. In the functional context, <em>setting</em> means to produce a new copy from an old copy, with some modification applied. Optics allow zooming into small parts of very large structures, and making pinpoint modifications, without having to deal with all the boilerplate involved in deconstructing and reconstructing every level of the data structure.</p>
<p><strong>Orthogonal</strong>. Roughly speaking, a set of operators is <em>orthogonal</em> when no operator performs the function of any other operator. This is the functional programming analogue of <em>Single Responsibility Principle</em> (SRP). More precisely, a set of operators <code class="language-plaintext highlighter-rouge">S</code> is <em>orthogonal</em> if there exists no other factoring of the operators <code class="language-plaintext highlighter-rouge">S1</code> such that any operator in <code class="language-plaintext highlighter-rouge">S</code> can be expressed as a composition of two or more operators in <code class="language-plaintext highlighter-rouge">S1</code>.</p>
<p><strong>Parametric Polymorphism</strong>. Sometimes called <em>generics</em>, <em>parametric polymorphism</em> is a feature of some programming languages that allows <em>universally quantifying</em> a function or a data type over one or more type parameters. Such <em>polymorphic functions</em> and <em>polymorphic data types</em> are said to be <em>parameterized</em> by their type parameters. Parametric polymorphism allows the creation of generic code, which works across many different data types, and generic data types, like collections. Parametrically polymorphic code must behave uniformly across all choice of type parameters, which allows a powerful way of reasoning about such code called <em>parametric reasoning</em>.</p>
<p><strong>Parametric Reasoning</strong>. <em>Parametric reasoning</em> is a type of reasoning that enables one to state facts about an implementation of a polymorphic function or polymorphic data type based only on type signatures. Parametric reasoning is typically accomplished using <em>flow analysis</em> on polymorphic declarations.</p>
<p><strong>Partial Function</strong>. A <em>partial function</em> is a function that is only defined for a subset of its domain. Partial functions can be modeled as total functions by expanding the codomain to include at least one new value, which indicates the function does not handle some input.</p>
<p><strong>Polymorphism</strong>. <em>Polymorphism</em> is a feature of programming languages that allows a variable or function to take on many different forms. Common types of polymorphism include <em>parametric polymorphism</em>, <em>subtype polymorphism</em>, and <em>ad hoc polymorphism</em>.</p>
<p><strong>Prisms</strong>. <em>Prisms</em> are <em>optics</em> that allow getting and setting terms in <em>sum types</em>. For example, a prism could allow getting and setting the <code class="language-plaintext highlighter-rouge">Left</code> term inside objects of type <code class="language-plaintext highlighter-rouge">Either[A, B]</code> (<code class="language-plaintext highlighter-rouge">Either a b</code>). In the functional context, <em>setting</em> a term in a sum means creating a new sum, given the new value for the term.</p>
<p><strong>Procedures</strong>. In programming languages, <em>procedures</em> are named entities that contain ordered sets of instructions for a machine to perform. Usually called <em>functions</em>, <em>procedures</em>, or <em>subroutines</em> in common parlance, procedures are not necessarily <em>functions</em> in the mathematical sense of the word, although all mathematical functions on values constitute valid procedures.</p>
<p><strong>Procedural Programming</strong>. <em>Procedural programming</em> is a style of programming in which solutions are constructed from the construction and invocation of procedures. All procedural programming is <em>imperative</em> in nature, although not all imperative programming is <em>procedural</em>.</p>
<p><strong>Product Type</strong>. A <em>product type</em> is the composition of <code class="language-plaintext highlighter-rouge">n</code> types, referred to as the <em>terms</em> of the product, together into a new type, such that a value of the product type contains a value from each of its terms. Structs, classes, records, and tuples are all examples of product types. A table is also an example of a product type, because each row in a table contains a value from each of its columns.</p>
<p><strong>Profunctor</strong>. A <em>profunctor</em> is type class defined over type constructors of two type parameters that describes polymorphic data types that form a <code class="language-plaintext highlighter-rouge">Functor</code> in one type parameter, and a <code class="language-plaintext highlighter-rouge">Contravariant</code> Functor in the other type parameter. Functions are examples of profunctors, which can be mapped on the output channel, and contramapped on the input channel. Profunctors can be thought of as a generalization of functions, which accept input and produce output, but most profunctors are not functions.</p>
<p><strong>Projection</strong>. A <em>projection</em> is a “lossy” mapping from one set to another set, such that if applied again to the output of itself, returns an equivalent result. An example is pulling the <em>age</em> field out of a <em>Person</em>, in which case repeated application produces the same <em>age</em> field. All extractions of terms from product types can be regarded as projections.</p>
<p><strong>Property-based Testing</strong>. <em>Property-based testing</em> is a method of testing <em>algebraic laws</em> by pseudo-randomly generating example values, and testing to see if the laws are satisfied. For example, property-testing could generate many integers <code class="language-plaintext highlighter-rouge">a</code>, <code class="language-plaintext highlighter-rouge">b</code>, and <code class="language-plaintext highlighter-rouge">c</code> to check if <code class="language-plaintext highlighter-rouge">(a + b) + c</code> is equal to <code class="language-plaintext highlighter-rouge">a + (b + c)</code>. Property-based testing cannot <em>prove</em> laws, but it can disprove them, as well as ensure there are no trivial law violations.</p>
<p><strong>Pure</strong>. A <em>pure</em> procedure combines inputs into an output, and does not interact with the world outside the function in any way that can be observed. All pure procedures are <em>deterministic</em>, but not all <em>deterministic</em> procedures are pure.</p>
<p><strong>Record</strong>. A <em>record</em> contains <code class="language-plaintext highlighter-rouge">n</code> fields, each accessed with some <em>label</em>, called the <em>name</em> of the field, and each with some associated type. All records are product types, although not all product types are records (for example, terms of tuples are accessed by ordinal values, not names).</p>
<p><strong>Recursive Data</strong>. <em>Recursive data</em> is any data type that appears as a value somewhere inside the data type. Common examples of recursive data include linked-lists and trees.</p>
<p><strong>Recursive Functions</strong>. <em>Recursive functions</em> are functions that call themselves. In functional programming, recursive functions allow iteration, like processing elements in a list or accepting socket connections.</p>
<p><strong>Recursion Schemes</strong>. <em>Recursions schemes</em> refers to any of the many different ways to traverse recursive data types, such as folding (catamorphisms) or unfolding (anamorphisms). Typically <em>recursion schemes</em> refers to the use of fixed-point data types to factor out recursion from data types, and minimize the boilerplate involved in creating different recursion schemes for different recursive data types. However, fundamentally, any recursive data type can be constructed, deconstructed, and transformed in any of the different schemes.</p>
<p><strong>Referential Transparency</strong>. <em>Referential transparency</em> is a property of functional programs in which any expression may be replaced by the value it computes without changing the behavior of the program. Referential transparency allows refactoring programs without changing their behavior, and is necessary (but not sufficient) for <em>local reasoning</em>.</p>
<p><strong>Reified</strong>. A <em>reified</em> construct is some non-value construct (such as a type, a field, or a package) that has been represented as a value. For example, reified generics store the type of generic type parameters as values at runtime. Similarly, lenses are a type of reified field, which provide a way to treat a field of a record as a first-class value, which can be passed around, stored, and transformed.</p>
<p><strong>Semigroup</strong>. A <em>semigroup</em> is a type class with a single binary operator, often called <em>append</em> or <em>combine</em>, that is <em>assocative</em>.</p>
<p><strong>Side-Effect</strong>. A <em>side-effect</em> occurs when evaluation of an expression does anything more than computing a value. Side-effects are interactions that can be observed from outside a procedure or expression. Common side-effects include database access, file access, network access, system calls, modifying mutable memory, or calling procedures that do any of the above.</p>
<p><strong>State</strong>. <em>State</em> refers to any data that is used to iteratively process information, either in a loop, or using recursion.</p>
<p><strong>Structural Typing</strong>. <em>Structural typing</em> is a method of defining types solely based on their structure. Two structural types are equal if and only if they have the same structure. Structural typing stands in contrast to <em>nominal typing</em>, in which two types that have the same structure are regarded as different if they have different names.</p>
<p><strong>Structure</strong>. The <em>structure</em> of a data type is the set of statements we know to be true about the data type. Parametrically polymorphic types (as well as <code class="language-plaintext highlighter-rouge">Any</code> in Scala) have <em>no</em> structure, but we can add structure by requiring these types have <em>instances</em> for one or more <em>type classes</em>.</p>
<p><strong>Subtyping</strong>. <em>Subtyping</em> is a method of defining types that permits “subset” relations, where one type is defined to be a subset or superset of another type. For example, all dogs are animals, and a type system with subtyping can recognize the fact that a <code class="language-plaintext highlighter-rouge">Dog</code> type is a subset of an <code class="language-plaintext highlighter-rouge">Animal</code> type.</p>
<p><strong>Sum Type</strong>. A <em>sum type</em> is the composition of <code class="language-plaintext highlighter-rouge">n</code> types together in an enumeration type, where each case of the enumeration is referred to as a <em>term</em> in the sum type. A value from a sum type contains a value from exactly one of its terms. For example, a value of a <code class="language-plaintext highlighter-rouge">Suite</code> enumeration is either <code class="language-plaintext highlighter-rouge">Hearts</code>, <code class="language-plaintext highlighter-rouge">Diamonds</code>, <code class="language-plaintext highlighter-rouge">Spades</code>, or <code class="language-plaintext highlighter-rouge">Clubs</code>. In Scala 2.x, enumerations are emulated with <code class="language-plaintext highlighter-rouge">sealed trait</code>, where the terms of the sum type are subtypes of the <code class="language-plaintext highlighter-rouge">sealed trait</code>.</p>
<p><strong>Tagless-Final</strong>. <em>Tagless-final</em> refers to a way of encoding the interpretation of a DSL using parametric polymorphism. In tagless-final, code written in a DSL is polymorphic in the data type used for interpretation. This polymorphic code is interpreted in different ways by instantiating it to concrete evaluation types. In Scala, <em>tagless-final</em> refers almost exclusively to using <em>indirection</em> to provide ad hoc operations across polymorphic effect types.</p>
<p><strong>Total</strong>. A <em>total</em> procedure returns an output for every input. Total procedures, unlike partial procedures, always terminate, and they never throw exceptions.</p>
<p><strong>Traversals</strong>. <em>Traversals</em> are <em>optics</em> that allow getting and setting elements in collection-like data structures.</p>
<p><strong>Type</strong>. A <em>type</em> is information that exists at compile-time, stored in the computer’s memory as the program is compiling. Types are used by the compiler to describe the structure of variables and functions, to ensure programs are well-defined and consistent. Informally, a type can be regarded as a mathematical set of values; the type <code class="language-plaintext highlighter-rouge">Boolean</code>, for example, contains the values <em>true</em> and <em>false</em>. To ascribe a variable a type in a programming language is to assert the value is a member of the set described by the type, and the act of constructing such a value can be regarded as a proof that the assertion is correct.</p>
<p><strong>Type Class</strong>. A <em>type class</em> is a language-level encoding of an <em>abstraction</em>. Type classes can be viewed as functions from some type to an <em>algebraic structure</em> for that type. For example, an <code class="language-plaintext highlighter-rouge">Ord</code> type class might encode the algebraic structure of total ordering, and might allow us to access ordering operations for types that have this algebraic structure.</p>
<p><strong>Type Constructors</strong>. A <em>type constructor</em> is a type that is itself parameterized by other types. To construct a type, the type constructor must be <em>fully applied</em> to all of its parameters, which then results in a type. For example, <code class="language-plaintext highlighter-rouge">List</code> (<code class="language-plaintext highlighter-rouge">[]</code>) is a type constructor. Given a type, such as <code class="language-plaintext highlighter-rouge">Int</code> (<code class="language-plaintext highlighter-rouge">Integer</code>), then “passing” that type to the <code class="language-plaintext highlighter-rouge">List</code> type constructor then constructs a type: namely, the type of lists with that specified element type. The structure of a type constructor is described by its <em>kind</em>.</p>
<p><strong>Universal Quantification</strong>. <em>Universal quantification</em> asserts that a statement holds for all possible choices of a given variable. All <em>parametrically polymorphic</em> functions and data types universally quantify over their type parameters, asserting the function or data type has an implementation for all possible choices of the type parameters. For example, the function <code class="language-plaintext highlighter-rouge">def empty[A]: List[A]</code> (<code class="language-plaintext highlighter-rouge">empty :: [a]</code>) universally quantifies over the type parameter <code class="language-plaintext highlighter-rouge">A</code>, asserting that for any element type, the function can produce a list of that element type (namely, the empty list).</p>
<p><strong>Universal Type</strong>. A <em>universal type</em> is any type variable used in <em>univeral quantification</em>. In programming languages, universal types are introduced by <em>parametrically polymorphic</em> function and data type declarations.</p>
<p><strong>Value</strong>. A <em>value</em> is information that exists at runtime, stored in the computer’s memory as the program is executing. Values are used to hold and transmit information within a program, and across process boundaries, to the operating system, file system, network, and beyond. In programming languages, values are constructed from literals or from <em>expressions</em>.</p>
<h2 id="faq">FAQ</h2>
<p><em>Question</em>: Can you define some other term that I find confusing?</p>
<p><em>Answer</em>: Sure, send it on and I’ll add it to the glossary!</p>
<p><em>Question</em>: Wikipedia doesn’t agree with your definitions.</p>
<p><em>Answer</em>: Oh no!</p>
<p><em>Question</em>: I don’t agree with your definitions.</p>
<p><em>Answer</em>: Ok.</p>
<p><em>Question</em>: Change your definitions to what I want!</p>
<p><em>Answer</em>: Nope. But if you have ideas to improve these definitions, please share your feedback!</p>
<p><a href="https://degoes.net/articles/fp-glossary">A Glossary of Functional Programming</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on December 22, 2019.</p>https://degoes.net/articles/tagless-horror2019-06-18T00:00:00-06:002019-06-18T00:00:00-06:00John A De Goeshttps://degoes.netjohn@degoes.net<p>Tagless-final is a technique originally used to <a href="https://okmij.org/ftp/tagless-final/index.html">embed domain-specific languages</a> into a host language, without the use of Generalized Algebraic Data Types.</p>
<p>In the Haskell community, <em>tagless-final</em> still refers to a way of creating polymorphic programs in a custom DSL that are interpreted by instantiating them to a concrete data type. In the Scala community, however, tagless-final is used almost exclusively for monadic, effectful DSLs. Usage of the term in Scala is closest to what Haskeller’s mean by <em>MTL-style</em>, but without the algebraic laws that govern MTL type classes.</p>
<p>In Scala, tagless-final has become almost synonymous with types of kind <code class="language-plaintext highlighter-rouge">* -> *</code>, leading to the infamous <code class="language-plaintext highlighter-rouge">F[_]</code> higher-kinded type parameter that is so pervasively associated with the phrase <em>tagless-final</em>.</p>
<p>In this post, I will argue that the many claims made about tagless-final in Scala are not <em>entirely</em> true, and that the actual benefits of tagless-final come mostly from <em>discipline</em>, not from so-called <em>effect polymorphism</em>.</p>
<p>After this detailed analysis, I will conclude the post by providing a list of concrete recommendations for developers who are building functional Scala libraries and applications.</p>
<h2 id="tagless-final-101">Tagless-Final 101</h2>
<p>In Scala, tagless-final involves creating <em>type classes</em>, which describe capabilities of a generic effect <code class="language-plaintext highlighter-rouge">F[_]</code>.</p>
<p><em>Note: There is an alternate, and (I’d argue) superior encoding of tagless-final that involves not type classes, but effect-polymorphic interfaces, which are passed as ordinary parameters; but this alternate encoding doesn’t substantially change my arguments, so I won’t discuss it here.</em></p>
<p>For example, we can create the following type class to describe console-related capabilities of some effect <code class="language-plaintext highlighter-rouge">F[_]</code>:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">trait</span> <span class="nc">Console</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]]</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">putStrLn</span><span class="o">(</span><span class="n">line</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span>
<span class="k">val</span> <span class="nv">getStrLn</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>
<span class="o">}</span></code></pre></figure>
<p>Or, we could create the following type class to describe persistence capabilities for <code class="language-plaintext highlighter-rouge">User</code> objects:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">trait</span> <span class="nc">UserRepository</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]]</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">getUserById</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">UserID</span><span class="o">)</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">User</span><span class="o">]</span>
<span class="k">def</span> <span class="nf">getUserProfile</span><span class="o">(</span><span class="n">user</span><span class="k">:</span> <span class="kt">User</span><span class="o">)</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">UserProfile</span><span class="o">]</span>
<span class="k">def</span> <span class="nf">updateUserProfile</span><span class="o">(</span><span class="n">user</span><span class="k">:</span> <span class="kt">User</span><span class="o">,</span> <span class="n">profile</span><span class="k">:</span> <span class="kt">UserProfile</span><span class="o">)</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span>
<span class="o">}</span></code></pre></figure>
<p>These type classes allow us to create methods that are <em>polymorphic</em> in the effect type <code class="language-plaintext highlighter-rouge">F[_]</code>, but which have access to required capabilities. For example, we could describe a program that uses the <code class="language-plaintext highlighter-rouge">Console</code> interface as follows:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">consoleProgram</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Console</span><span class="o">]</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span>
<span class="n">implicitly</span><span class="o">[</span><span class="kt">Console</span><span class="o">[</span><span class="kt">F</span><span class="o">]].</span><span class="py">putStrLn</span><span class="o">(</span><span class="s">"Hello World!"</span><span class="o">)</span></code></pre></figure>
<p>Combined with type classes like <code class="language-plaintext highlighter-rouge">Monad</code> (which allows chaining effects), we can build entire programs using the tagless-final approach:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">consoleProgram</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Console:</span> <span class="kt">Monad</span><span class="o">]</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">val</span> <span class="nv">console</span> <span class="k">=</span> <span class="n">implicitly</span><span class="o">[</span><span class="kt">Console</span><span class="o">[</span><span class="kt">F</span><span class="o">]]</span>
<span class="k">import</span> <span class="nn">console._</span>
<span class="k">for</span> <span class="o">{</span>
<span class="k">_</span> <span class="k"><-</span> <span class="nf">putStrLn</span><span class="o">(</span><span class="s">"What is your name?"</span><span class="o">)</span>
<span class="n">name</span> <span class="k"><-</span> <span class="n">getStrLn</span>
<span class="k">_</span> <span class="k"><-</span> <span class="nf">putStrLn</span><span class="o">(</span><span class="n">s</span><span class="s">"Hello, $name, good to meet you!"</span><span class="o">)</span>
<span class="o">}</span> <span class="k">yield</span> <span class="n">name</span>
<span class="o">}</span></code></pre></figure>
<p>Because such programs are polymorphic in the effect type <code class="language-plaintext highlighter-rouge">F[_]</code>, we can <em>instantiate</em> these polymorphic programs to any concrete effect type that provides whatever they require.</p>
<p>For example, if we are using ZIO <code class="language-plaintext highlighter-rouge">Task</code> (a type alias for <code class="language-plaintext highlighter-rouge">ZIO[Any, Throwable, A]</code>), we can instantiate our program to this concrete effect type with the following code snippet:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">taskProgram</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="n">consoleProgram</span><span class="o">[</span><span class="kt">Task</span><span class="o">]</span></code></pre></figure>
<p>Typically, the <em>instantiation</em> of a polymorphic tagless-final value to a concrete effect type is deferred as long as possible, preferrably to the entry points of the application or test suite.</p>
<p>With this introduction, let’s talk about the reasons you might want to use tagless-final…the <em>pitch</em> for the tagless-final technique, if you will.</p>
<h2 id="the-pitch-for-tagless-final">The Pitch for Tagless-Final</h2>
<p>Tagless-final has a seductive pitch that appeals to every aspiring functional programmer.</p>
<p>In the functional programming community, we are taught (with good reason) that monomorphic code can’t be reused much, and leads to more bugs.</p>
<p>We are taught that generic code not only enables reuse, but it pushes more information into the types, where the compiler can help us verify and maintain correctness.</p>
<p>We are taught the <em>principle of least power</em>, which tells us that our functions should require as little as necessary to do their job.</p>
<p>I have helped develop, motivate, and teach these and other principles in my <em>Functional Scala</em> workshops, helping train new generations of Scala developers in the functional way of thinking and developing software.</p>
<p>In this context, functional programmers are <em>primed</em> for the tagless-final pitch; I know this, because I have <em>given</em> the tagless-final pitch, and even helped <em>craft</em> its modern day incarnation.</p>
<p>In one video, I unintentionally convinced <a href="https://www.youtube.com/watch?v=sxudIMiOo68">several companies to adopt tagless-final</a>, despite an explicit disclaimer stating the techniques would be overkill for many applications!</p>
<p>In the next few sections, I’m going to give you this pitch, and try to convince you that tagless-final is the best thing ever. Moreover, I’m going to use only arguments that have an element of truth.</p>
<p>Ready? Here we go!</p>
<h3 id="1-effect-type-indirection">1. Effect Type Indirection</h3>
<p>As of this writing, there are <a href="/articles/zio-cats-effect">several mainstream effect types</a>, including ZIO, Monix, and Cats IO, all of which ship with <a href="https://github.com/typelevel/cats-effect">Cats Effect</a> instances, and which can be used more or less interchangeably in libraries like FS2, Doobie, and http4s.</p>
<p>Tagless-final lets you insulate your code from the decision of <em>which</em> effect type to use. Rather than pick one of these concrete implementations, using tagless-final lets you write <em>effect type-agnostic</em> code, which can be <em>instantiated</em> to any concrete effect type that provides Cats Effect instances.</p>
<p>For example, our preceding console program can just as easily be instantiated to Cats IO:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">ioProgram</span> <span class="k">=</span> <span class="n">consoleProgram</span><span class="o">[</span><span class="kt">cats.effect.IO</span><span class="o">]</span></code></pre></figure>
<p>With tagless-final, you can defer the decision of which effect type to use <em>indefinitely</em> (or at least, to the edge of your program), isolating your application from changes in an evolving ecosystem.</p>
<p>Tagless-final lets you future-proof your code!</p>
<h3 id="2-effect-testability">2. Effect Testability</h3>
<p>Tagless-final, because it provides a strong layer of indirection between your application, and the concrete effect type that models effects, enables your code to be fully testable.</p>
<p>In the preceding console implementation, it is easy to define a test instance of the <code class="language-plaintext highlighter-rouge">Console</code> type class:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">ConsoleData</span><span class="o">(</span><span class="n">input</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">],</span> <span class="n">output</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span>
<span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">ConsoleTest</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">run</span><span class="k">:</span> <span class="kt">ConsoleData</span> <span class="o">=></span> <span class="o">(</span><span class="nc">ConsoleData</span><span class="o">,</span> <span class="n">A</span><span class="o">))</span>
<span class="k">object</span> <span class="nc">ConsoleTest</span> <span class="o">{</span>
<span class="k">implicit</span> <span class="k">def</span> <span class="nf">ConsoleConsoleTest</span><span class="k">:</span> <span class="kt">Console</span><span class="o">[</span><span class="kt">ConsoleTest</span><span class="o">]</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">Console</span><span class="o">[</span><span class="kt">ConsoleTest</span><span class="o">]</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">putStrLn</span><span class="o">(</span><span class="n">line</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">ConsoleTest</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span>
<span class="nc">ConsoleTest</span><span class="o">(</span><span class="n">data</span> <span class="k">=></span> <span class="o">(</span><span class="nv">data</span><span class="o">.</span><span class="py">copy</span><span class="o">(</span><span class="n">output</span> <span class="k">=</span> <span class="n">line</span> <span class="o">::</span> <span class="nv">data</span><span class="o">.</span><span class="py">output</span><span class="o">),</span> <span class="o">()))</span>
<span class="k">val</span> <span class="nv">getStrLn</span><span class="k">:</span> <span class="kt">ConsoleTest</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span>
<span class="nc">ConsoleTest</span><span class="o">(</span><span class="n">data</span> <span class="k">=></span> <span class="o">(</span><span class="nv">data</span><span class="o">.</span><span class="py">copy</span><span class="o">(</span><span class="n">input</span> <span class="k">=</span> <span class="nv">data</span><span class="o">.</span><span class="py">input</span><span class="o">.</span><span class="py">drop</span><span class="o">(</span><span class="mi">1</span><span class="o">)),</span> <span class="nv">data</span><span class="o">.</span><span class="py">input</span><span class="o">.</span><span class="py">head</span><span class="o">))</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></figure>
<p>Now, assuming we define an appropriate <code class="language-plaintext highlighter-rouge">Monad</code> instance for our data type (which is easy to do!), we can instantiate our polymorphic <code class="language-plaintext highlighter-rouge">consoleProgram</code> to the new type:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">testProgram</span> <span class="k">=</span> <span class="n">consoleProgram</span><span class="o">[</span><span class="kt">ConsoleTest</span><span class="o">]</span></code></pre></figure>
<p>Tada! We can now unit test our console program with fast, deterministic unit tests, thereby reaping the full testability benefits of pure functional programming.</p>
<h3 id="3-effect-parametric-reasoning">3. Effect Parametric Reasoning</h3>
<p><em>Parametric reasoning</em> in statically-typed functional programming circles refers to the ability for us to reason generically about the implementation of a polymorphic function merely by looking at its type.</p>
<p>For example, there is one possible implementation of the following function:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">identity</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">a</span><span class="k">:</span> <span class="kt">A</span><span class="o">)</span><span class="k">:</span> <span class="kt">A</span> <span class="o">=</span> <span class="o">???</span></code></pre></figure>
<p>(Assuming no reflection, exceptions, or use of <code class="language-plaintext highlighter-rouge">null</code>.)</p>
<p>Parametric reasoning allows us to save time when we are studying code. We can look at the types of a function, and even if we don’t know the exact implementation, we can place <em>bounds</em> on what the function can do.</p>
<p>Parametric reasoning lets us more quickly and more reliably understand code bases, which is critical for safe maintenance of those code bases in response to new and changing business requirements.</p>
<p>Moreover, parametric reasoning can reduce the need for unit testing: whatever is guaranteed by the type, does not need to be tested by unit tests. Types prove universal properties across all values, so they are strictly more powerful than tests, which prove existential properties across a few values.</p>
<p>Since tagless-final is an example of (higher-kinded) parametric polymorphism, we can leverrage parametric polymorphism for effect types too.</p>
<p>For example, take the following code snippet:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">consoleProgram</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Console:</span> <span class="kt">Applicative</span><span class="o">]</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="o">???</span></code></pre></figure>
<p>Although we don’t know what the implementation of this function is without looking, we know that it requires <code class="language-plaintext highlighter-rouge">F[_]</code> to provide both <code class="language-plaintext highlighter-rouge">Console</code> and <code class="language-plaintext highlighter-rouge">Applicative</code> instances.</p>
<p>Because <code class="language-plaintext highlighter-rouge">F[_]</code> is only <code class="language-plaintext highlighter-rouge">Applicative</code> and not <code class="language-plaintext highlighter-rouge">Monad</code>, we know that although <code class="language-plaintext highlighter-rouge">consoleProgram</code> can have a sequential chain of console effects, no subsequent effect can depend on the runtime value of a predecessor effect (that capability would require <code class="language-plaintext highlighter-rouge">bind</code> from <code class="language-plaintext highlighter-rouge">Monad</code>).</p>
<p>We would not be surprised if we saw the implementation were as follows:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">consoleProgram</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Console:</span> <span class="kt">Applicative</span><span class="o">]</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">val</span> <span class="nv">console</span> <span class="k">=</span> <span class="n">implicitly</span><span class="o">[</span><span class="kt">Console</span><span class="o">[</span><span class="kt">F</span><span class="o">]]</span>
<span class="k">import</span> <span class="nn">console._</span>
<span class="nf">putStrLn</span><span class="o">(</span><span class="s">"What is your name?"</span><span class="o">)</span> <span class="o">*></span> <span class="n">getStrLn</span>
<span class="o">}</span></code></pre></figure>
<p>By constraining the capabilities of the data type <code class="language-plaintext highlighter-rouge">F[_]</code>, tagless-final lets us reason about our effectful programs parametrically, which lets us spend less time studying code, and less time writing unit tests.</p>
<h3 id="the-closer">The Closer</h3>
<p>As we have seen, tagless-final is an incredible asset to functional Scala programmers everwhere.</p>
<p>Not only does it enable abstraction over effects, so programmers can change their mind about which effect type to use, but the technique gives us full testability of effectful programs, and helps us reason about our effectful programs in new ways, which can reduce the need for studying code and cut down on unit tests.</p>
<h2 id="the-fine-print">The Fine Print</h2>
<p>Sold yet on tagless-final? I am!!! Well, <em>somewhat</em>, anyway.</p>
<p>There’s an element of truth in <em>every</em> argument that I’ve made. Although as you might suspect, a more nuanced, sophisticated look reveals a less positive picture of tagless-final.</p>
<p>Let’s take a look at all the fine print in the next few sections.</p>
<h3 id="1-premature-indirection">1. Premature Indirection</h3>
<p>It is absolutely true that adding a layer of indirection over an effect type can reduce the cost of switching to a different effect type, assuming similar underlying semantics.</p>
<p>It is also true that adding a layer of indirection over Spark, Akka https, Slick, or a database-specific dialect of SQL, might reduce the cost of switching to different technologies.</p>
<p>In my experience, however, the attempt to proactively add layers of indirection without a clear business mandate to do so, simply to mitigate the <em>possible</em> cost of future change, is an example of <em>premature indirection</em>.</p>
<p>Premature indirection rarely pays for itself.</p>
<p>In most cases, overall application costs would be <em>substantially lower</em> picking a specific technology, and then, if needs change, simply refactoring the code to a new technology.</p>
<p>The cost of refactoring from one effect type to another is not related to the cost of changing types from <code class="language-plaintext highlighter-rouge">IO[_]</code> to <code class="language-plaintext highlighter-rouge">Task[_]</code>, or swapping one set of methods for another. Rather, it is related to the <em>semantic</em> differences between operations on these effect types. Yet a layer of indirection helps us only when semantic differences are relatively small. If they are small, then the cost of refactoring is relatively low.</p>
<p>A refactoring from one effect type to another only needs to happen once, and only if actually necessary (which it might not be). But the cost of coding to a layer of indirection has to be paid indefinitely, and it must be paid regardless of whether or not the indirection will ever be used.</p>
<p>Beyond the cost of coding to an indirection layer that may never be used, there are substantial <em>opportunity costs</em> to premature indirection. In the case of ZIO, for example, the core effect type has hundreds of additional operations that are not available on a polymorphic <code class="language-plaintext highlighter-rouge">F[_]</code>.</p>
<p>These methods, like many additional methods on Monix Task, are discoverable by IDEs; their types guide users to correct solutions; they have great inline Scaladoc; error messages are concrete and actionable; and type-inference is nearly flawless.</p>
<p>If you <em>commit to not committing</em>, you’re stuck with the weakest <code class="language-plaintext highlighter-rouge">F[_]</code>, which means much of the power of an effect system remains inaccessible. The frustration of knowing a method is right <em>there</em>, but just out of reach, has prompted many to introduce custom type classes designed to access specific features of the underlying effect type.</p>
<p>The opportunity cost is even greater for ZIO than Monix, because ZIO features polymorphic reader and error effects, which provide new operations and allow parametric reasoning about dependencies and error handling, and which are currently unsupported in Cats Effect (the leading library for effect indirection).</p>
<p>In my opinion, only library authors have a compelling argument for effect type indirection. In order to maximize market share (which is critical for the adoption, retention, and growth of open source libraries), they need to support all major effect types, or risk losing market share to the libraries that do.</p>
<p>While this is a compelling argument for open source libraries, it is completely inapplicable to the closed source applications that make up the majority of Scala software development.</p>
<h3 id="2-untestable-effects">2. Untestable Effects</h3>
<p>Tagless-final provides a <em>path</em> to testability, but tagless-final programs are not <em>inherently</em> testable. In fact, they are testable <em>only to the degree their tagless-final type classes are testable</em>.</p>
<p>For example, while we can create a testable version of the <code class="language-plaintext highlighter-rouge">Console</code> type class, many others, including popular type classes in the tagless-final community, are inherently <em>untestable</em>.</p>
<p>The majority of applications written in the tagless-final style make heavy use of a type class in Cats Effect called <code class="language-plaintext highlighter-rouge">Sync</code>, or its more powerful versions, including <code class="language-plaintext highlighter-rouge">Async</code>, <code class="language-plaintext highlighter-rouge">LiftIO</code>, <code class="language-plaintext highlighter-rouge">Concurrent</code>, <code class="language-plaintext highlighter-rouge">Effect</code> and <code class="language-plaintext highlighter-rouge">ConcurrentEffect</code>.</p>
<p>These type classes are designed to capture side-effects—for example, random number generation, API calls, and database queries. Any code that utilizes these type classes, or others like them, cannot be unit tested even in theory, because it interacts with partial, non-deterministic, and side-effecting code.</p>
<p>While such code can be tested with integration and system tests, <em>any code at all</em> can be tested with integration and system tests, including the worst possible procedural code!</p>
<p>Ultimately, the testability of tagless-final programs requires they <em>code to an interface, not an implementation</em>. Yet, if applications follow this principle, they can be tested even <em>without</em> tagless-final!</p>
<p>Here is a monomorphic version of the <code class="language-plaintext highlighter-rouge">Console</code> type class, for example:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">trait</span> <span class="nc">Console</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">putStrLn</span><span class="o">(</span><span class="n">line</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span>
<span class="k">val</span> <span class="nv">getStrLn</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>
<span class="o">}</span></code></pre></figure>
<p>If your program uses this interface to perform console input/output, then it can be tested, even with a monomorphic effect type. The key insight is that your program must be <em>written to an interface</em>, rather than a concrete implementation. Unfortunately, numerous widely-used tagless-final interfaces (like <code class="language-plaintext highlighter-rouge">Sync</code>, <code class="language-plaintext highlighter-rouge">Async</code>, <code class="language-plaintext highlighter-rouge">LiftIO</code>, <code class="language-plaintext highlighter-rouge">Concurrent</code>, <code class="language-plaintext highlighter-rouge">Effect</code>, and <code class="language-plaintext highlighter-rouge">ConcurrentEffect</code>) encourage you to code to an <em>implementation</em>.</p>
<p>Using tagless-final doesn’t provide any inherent benefits to testability. The testability of your application is completely orthogonal to its use of tagless-final, and comes down to whether or not you follow best practices—which you can do <em>with</em> or <em>without</em> tagless-final.</p>
<h3 id="3-no-effect-polymorphic-reasoning">3. No Effect Polymorphic Reasoning</h3>
<p>The most insidious claim that tagless-final makes is that it provides <em>effect polymorphic reasoning</em>.</p>
<p>According to this claim—which is not entirely without merit—we can look at a polymorphic function signature, and know the effects it performs merely by examining constraints.</p>
<p>Previously, we looked at the following example:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">consoleProgram</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Console:</span> <span class="kt">Applicative</span><span class="o">]</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="o">???</span></code></pre></figure>
<p>I argued we could know something about what this function does by observing its use of <code class="language-plaintext highlighter-rouge">Applicative</code> and <code class="language-plaintext highlighter-rouge">Console</code>. In theory, these constraints provides us the ability to <em>partially</em> understand the behavior of the function without examining the implementation.</p>
<p>Unfortunately, this argument rests on a premise that is <em>false</em>. Namely, it rests on the premise that implicit parameters somehow <em>constrain</em> the side-effects executed or modeled by the function.</p>
<p>Scala does not track side-effects, no matter how much we wish otherwise. We can perform console effects anywhere, even without the provided <code class="language-plaintext highlighter-rouge">Console[F]</code>, as shown in the following snippet:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">consoleProgram</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Applicative</span><span class="o">]</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="nf">println</span><span class="o">(</span><span class="s">"What is your name?"</span><span class="o">)</span>
<span class="k">val</span> <span class="nv">name</span> <span class="k">=</span> <span class="nv">scala</span><span class="o">.</span><span class="py">io</span><span class="o">.</span><span class="py">StdIn</span><span class="o">.</span><span class="py">readLine</span><span class="o">()</span>
<span class="nc">Applicative</span><span class="o">[</span><span class="kt">F</span><span class="o">].</span><span class="py">pure</span><span class="o">(</span><span class="n">name</span><span class="o">)</span>
<span class="o">}</span></code></pre></figure>
<p>Not only can we perform any effects anywhere inside the body of the method without even so much as a compiler warning, but we can embed these effects into any <code class="language-plaintext highlighter-rouge">Applicative</code> functor, even one with a strict definition of <code class="language-plaintext highlighter-rouge">pure</code> / <code class="language-plaintext highlighter-rouge">point</code>:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">consoleProgram</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Applicative</span><span class="o">]</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span>
<span class="nc">Applicative</span><span class="o">[</span><span class="kt">F</span><span class="o">].</span><span class="py">pure</span><span class="o">(()).</span><span class="py">map</span> <span class="o">{</span> <span class="k">_</span> <span class="k">=></span>
<span class="k">val</span> <span class="nv">_</span> <span class="k">=</span> <span class="nf">println</span><span class="o">(</span><span class="s">"What is your name?"</span><span class="o">)</span>
<span class="nv">scala</span><span class="o">.</span><span class="py">io</span><span class="o">.</span><span class="py">StdIn</span><span class="o">.</span><span class="py">readLine</span><span class="o">()</span>
<span class="o">}</span></code></pre></figure>
<p>This is legal, type-safe Scala code, and variations of this code appear pervasively in real world Scala. Not even the most aggressive IntelliJ IDEA lint settings will report any problems with this code snippet (except perhaps a warning about the use of higher-kinded types).</p>
<p>In tagless-final, “constraints”—parameter values like <code class="language-plaintext highlighter-rouge">Applicative[F]</code> or <code class="language-plaintext highlighter-rouge">Console[F]</code>—do not actually constrain, they <em>unconstrain</em>, giving us more ways of constructing values. And in Scala, because side-effects are already <em>totally unconstrained</em>, adding more values to parameter lists doesn’t change anything.</p>
<p>Stated more forcefully and for Scala code, <strong>effect parametric reasoning is a lie</strong>.</p>
<p>Now, as highly-skilled and highly-disciplined functional programmers, we can agree amongst ourselves to reject merging such code into our projects. We can inspect every line of code in our application to ensure that side-effects are captured only in “appropriate” places, where we all agree on some definition of <em>appropriate</em>.</p>
<p>A social contract, powered by good-will, skill, discipline, and indoctrination of new hires, can be quite useful. Nearly <em>all</em> best practices are social contracts, and they undoubtedly help us write better code. But social contracts should not be confused with compile-time constraints.</p>
<p>If we <em>assume</em> a working social contract to restrict side-effects in Scala, then we can assume restrictions on the effects in the preceding definition of <code class="language-plaintext highlighter-rouge">consoleProgram</code>, but these restrictions do not come from effect polymorphism (which does not exist in a language without effect tracking!). Rather, the restrictions come from diligence and discipline in enforcing the social contract—the line-by-line review of every new pull request.</p>
<p>If we are going to rely on a social contract to give us reasoning benefits, however, then we should consider other social contracts, such as <em>always code to an interface, not an implementation</em>.</p>
<p>Other social contracts can give us the exact same “reasoning benefits”, but with potentially better ergonomics. Stated differently, “reasoning benefits” is not a reason to prefer tagless-final over any other approach, because those benefits derive only from discipline (not from effect polymorphism), and discipline works for other approaches too.</p>
<p>These are hard limits on <em>effect parametric polymorphism</em> in Scala, which—short of inventing a compiler plug-in that overlays a new, effect-tracked language onto Scala—cannot be circumvented.</p>
<h3 id="4-sync-bloat">4. Sync Bloat</h3>
<p>Even if we ignore the fact that effect parametric polymorphism doesn’t exist in Scala (reasoning benefits come from discipline, not from tagless-final), we have a major problem that I term <em>sync bloat</em>.</p>
<p>In Scala, the Cats Effect type class hierarchy provides many type classes that are <em>explicitly designed</em> to capture side-effecting code. These type classes include <code class="language-plaintext highlighter-rouge">Sync</code>, <code class="language-plaintext highlighter-rouge">Async</code>, <code class="language-plaintext highlighter-rouge">LiftIO</code>, <code class="language-plaintext highlighter-rouge">Concurrent</code>, <code class="language-plaintext highlighter-rouge">Effect</code>, and <code class="language-plaintext highlighter-rouge">ConcurrentEffect</code>.</p>
<p>Methods that require one of these type classes can literally do <em>anything</em> they want, without constraints, even assuming a working social contract. These methods nullify the power of discipline, depriving us of any benefits to reasoning and testability, resulting in opaque blobs of side-effecting, untestable procedural code.</p>
<p>Nearly all tagless-final code (including some of the <a href="https://github.com/slamdata/quasar/search?q=Sync&unscoped_q=Sync">best open source functional Scala I know of</a>) makes liberal use of these type classes, freely embedding side-effects in numerous methods sprawled across the code base.</p>
<p>In a perfect world, perhaps programmers would create hundreds or thousands of fine-grained, testable type classes to represent separate concerns. But in the real world, the vast majority of programmers using tagless-final (even high-skilled, expert-level functional programmers!) are not doing this. Instead, they’re requiring type classes like <code class="language-plaintext highlighter-rouge">Sync</code> that encourage embedding arbitrary side-effects everywhere.</p>
<p>The pervasive phenomenon of <em>sync bloat</em> means that even if we have a working social contract, powered by discipline and diligence, we <em>still</em> aren’t gaining any benefits of effect parametricity.</p>
<h3 id="5-fake-abstraction">5. Fake Abstraction</h3>
<p>As I teach in <em>Functional Scala</em>, abstractions are sets of operations that satisfy algebraic laws, encoded using type classes. Abstractions allow us to describe common structure across a range of distinct data types.</p>
<p>Abstraction is the means by which we can write <em>principled</em> polymorphic code.</p>
<p>Without lawful operations, there is no abstraction, only layers of indirection masquerading as abstraction.</p>
<p>As practiced in Scala, tagless-final encourages a form of <em>fake abstraction</em>. To see this, let’s take a closer look at the <code class="language-plaintext highlighter-rouge">Console[F]</code> type class defined previously:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">trait</span> <span class="nc">Console</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]]</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">putStrLn</span><span class="o">(</span><span class="n">line</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span>
<span class="k">val</span> <span class="nv">getStrLn</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>
<span class="o">}</span></code></pre></figure>
<p>This trait has two operations, called <code class="language-plaintext highlighter-rouge">putStrLn</code> and <code class="language-plaintext highlighter-rouge">getStrLn</code>. We can provide implementations of these two different operations for many different data types.</p>
<p>Unfortunately, these operations satisfy no algebraic laws—none whatsoever! This means when we are writing polymorphic code, we have no way to reason generically about <code class="language-plaintext highlighter-rouge">putStrLn</code> and <code class="language-plaintext highlighter-rouge">getStrLn</code>.</p>
<p>For all we know, these operations could be launching threads, creating or deleting files, running a large number of individual side-effects in sequence, and so on.</p>
<p>This contrasts quite dramatically with type classes like <code class="language-plaintext highlighter-rouge">Ord</code>, which we can use to build a generic sorting algorithm that will <em>always</em> be correct if the algebraic laws of <code class="language-plaintext highlighter-rouge">Ord</code> are satisfied for the element type (<em>this</em> is principled functional programming).</p>
<p>The implications of this are profound. Consider the following type signature:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">consoleProgram</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Console:</span> <span class="kt">Applicative</span><span class="o">]</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="o">???</span></code></pre></figure>
<p>Because <code class="language-plaintext highlighter-rouge">Console[F]</code> is lawless, we cannot reason generically about the side-effects modeled by the returned <code class="language-plaintext highlighter-rouge">F[Unit]</code>. These side-effects are unconstrained precisely because there are no algebraic laws that govern the operations provided to us by <code class="language-plaintext highlighter-rouge">Console[F]</code>.</p>
<p>Two different implementations of <code class="language-plaintext highlighter-rouge">Console[F]</code> could do totally different things, without breaking any laws, so we cannot actually reason <em>generically</em> about the correctness of code using <code class="language-plaintext highlighter-rouge">Console[F]</code>. Our code may be correct for some <code class="language-plaintext highlighter-rouge">Console[F]</code>, and incorrect for other <code class="language-plaintext highlighter-rouge">Console[F]</code>.</p>
<p>Keep in mind that <code class="language-plaintext highlighter-rouge">Console[F]</code> is just a toy example. A realistic tagless-final type class would be far larger and more complex, containing numerous operations, which are ad hoc, unlawful, and impossible to reason about generically. If we write code using this type class, it will be highly coupled to unspecified implementation details.</p>
<p><em>Real</em> generic reasoning applies only to <em>real</em> abstractions, like <code class="language-plaintext highlighter-rouge">Monoid</code>, <code class="language-plaintext highlighter-rouge">Monad</code>, and type classes from functional programming. The moment we introduce ad hoc, unspecified, implementation-specific operations like <code class="language-plaintext highlighter-rouge">putStrLn</code> or <code class="language-plaintext highlighter-rouge">getStrLn</code>, we can no longer reason generically about the behavior of our code in any principled way.</p>
<p>What this means is that even if we grant that Scala has effect parametric polymorphism (which it doesn’t!), and even if we assume that developers won’t use type classes like <code class="language-plaintext highlighter-rouge">Sync</code> (which they will!), the ad hoc nature of tagless-final type classes means we don’t actually have <em>useful generic reasoning</em> across these type classes.</p>
<p>If only <code class="language-plaintext highlighter-rouge">Applicative[F]</code> is required, we can tell that <code class="language-plaintext highlighter-rouge">bind</code> is not used, but we cannot tell how many individual side-effects are chained together to make up a single operation like <code class="language-plaintext highlighter-rouge">putStrLn</code>. The limited reasoning we can do is <em>useless</em> (a parlor trick, at best!), precisely because of all the reasoning we <em>cannot</em> do.</p>
<p>Does <code class="language-plaintext highlighter-rouge">putStrLn</code> print a line of text to a console? Or does it launch a multithreaded <code class="language-plaintext highlighter-rouge">main</code> function with the whole application? Who knows. The types and laws don’t tell you anything.</p>
<p>Adding <code class="language-plaintext highlighter-rouge">Console[F]</code> to a type signatue is at best a prayer that whoever gives us a <code class="language-plaintext highlighter-rouge">Console[F]</code> will abide by an unspecified contract that has something to do with console input/output and will make our “generic” code work correctly.</p>
<p>Polymorphic code that we can reason about generically is an awesome benefit of statically-typed functional programming. But generic reasoning requires abstractions, which must always have algebraic laws. The moment we create fake abstractions (operations without laws), we aren’t doing principled functional programming anymore.</p>
<h2 id="the-tagless-final-hit-list">The Tagless-Final Hit List</h2>
<p>Tagless-final in Scala doesn’t entirely live up to the hype, as I’ve argued in this post:</p>
<ol>
<li><strong>Premature Indirection</strong>. For applications, using tagless-final to guard against the possibility of changing effect types is usually <em>overengineering</em> (premature indirection). Your application doesn’t add indirection layers for Akka Streams, Slick, Future, or even the dialect of SQL you’re using—because in most cases, the cost of building and maintaining that layer of indirection is <em>unending</em>. In the case of tagless-final effects, you also deprive yourself of many lawful operations and added type safety.</li>
<li><strong>Untestable Effects</strong>. Many popular tagless-final type classes encourage capturing side-effects. Even if we are disciplined and diligent, these type classes destroy our ability to reason about and unit test applications built using them. In the end, testability is not a property of tagless-final code; testability requires you code to an interface, which you can do with or without tagless-final.</li>
<li><strong>No Effect Parametric Polymorphism</strong>. Scala doesn’t constrain side-effects, and implicit parameters don’t change this. If you want to <em>constrain</em> side-effects, you need a <em>social contract</em>, enforced by discipline and diligence. In this case, reasoning benefits (such as they are) come only from painstaking review of every line of code, not from effect polymorphism. Yet other approaches that require discipline, like ensuring programmers only code to an interface, can provide similar benefits, but without the drawbacks of tagless-final.</li>
<li><strong>Sync Bloat</strong>. In Scala, real world tagless-final code tends not to use custom type classes, but rather, a few type classes that allow the unrestricted capture of side-effects. These <code class="language-plaintext highlighter-rouge">Sync</code> code bases do not confer any benefits to reasoning or testability, even if we are disciplined and diligent during code review. They combine the ceremony and boilerplate of tagless-final with the untestable, unreasonable nature of the worst procedural code.</li>
<li><strong>Fake Abstraction</strong>. Reasoning about generic code requires abstractions, which come equipped with algebraic laws. Algebraic laws precisely define common structure across different data types, and they let us reason generically about the correctness of polymorphic code. Yet most tagless-final type classes do not have any laws. Any code that uses “fake abstractions” is not actually generic, but instead, is closely wedded to unspecified implementation details.</li>
</ol>
<p>Beyond just not living up to the hype, tagless-final has a number of serious drawbacks:</p>
<ol>
<li>Tagless-final has significant pedagogical costs, because of the huge number of concepts it requires a team to master (parametric polymorphism, higher-kinded types, higher-kinded parametric polymorphism, type classes, higher-kinded type classes, the functor hierarchy, etc.).</li>
<li>Tagless-final has significant institutional costs, because of the level of ceremony and boilerplate involved (type classes, type class instances, instance summoners, syntax extensions, higher-kinded implicit parameter lists, non-inferrable types, compiler plug-ins, etc.).</li>
</ol>
<p>I’ve <a href="/articles/zio-environment">talked about these drawbacks</a> at length in the past, and I encourage readers to investigate for themselves the drawbacks of tagless-final in Scala.</p>
<h2 id="objections">Objections</h2>
<p>Some functional programmers, when presented with these drawbacks, immediately counter with the objection that <em>other approaches</em> (including the <a href="/articles/zio-environment">reader monad</a>) can’t constrain side-effects in Scala, either.</p>
<p>This is true, but also entirely beside the point.</p>
<p>There are <em>zero</em> approaches to statically constraining effects in Scala, because Scala cannot statically constrain effects. This means that when comparing two different approaches to managing effects in Scala, there is no dimension for “constraining effects”.</p>
<p>You can combine tagless-final with manual inspection of every line of code in an application, to ensure it satisfies the social contract that side-effects will only be executed and captured in “approved” places. This combination, which is powered by discipline (not effect polymorphism), provides both testability and reasoning benefits.</p>
<p>Similarly, you can combine the reader monad with manual inspection of every line of code in an application, to ensure it satisfies a social contract that logic will be written to interfaces, not implementations. As with tagless final, this combination is powered by discipline, and provides both testability and reasoning benefits.</p>
<p>Both tagless-final and the reader monad (and many other approaches) can indeed provide “guarantees” about effects, but it’s not really the <em>techniques</em> that are providing the guarantees, but the <em>programmers</em> who are manually reviewing and merging every line of code. These “guarantees” come from discipline, not from the Scala compiler.</p>
<p>Take the following snippet of tagless-final:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">console</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Monad:</span> <span class="kt">Console</span><span class="o">]</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span> <span class="o">???</span></code></pre></figure>
<p>The only way we “know” this function does not interact with databases or perform random number generation or do anything else is if we or our colleagues have inspected every line of code, and manually certified that it doesn’t break our social contract about where side-effects can be run and modeled.</p>
<p>The compiler doesn’t provide any assistance with this, and this “knowledge” is not related to effect polymorphism.</p>
<p>Similarly, take the following snippet of monomorphic code:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">console</span><span class="o">(</span><span class="n">c</span><span class="k">:</span> <span class="kt">Console</span><span class="o">)</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span> <span class="o">???</span></code></pre></figure>
<p>The only way we “know” this function does not interact with the database or perform random number generation or do anything else is if we or our colleagues have inspected every line of code, and manually certified that it doesn’t break our social contract about coding to interfaces, and not to implementations.</p>
<p>As with taglesss-final, the compiler doesn’t provide any assistance. Both these approaches and others to reasoning about side-effects come down to social contracts, powered by discipline and vigilence.</p>
<p>Another vague objection I have heard is that tagless-final is somehow more principled than other approaches, or that it somehow lends itself better to designing composable functional APIs.</p>
<p>In reality, principled, composable, functional code has everything to do with whether the operations of your domain satisfy algebraic laws. As practiced in Scala, tagless-final has no relation to principled functional programming.</p>
<p>The use of higher-kinded parametric polymorphism may provide an illusion of “rigor”, but it’s just a thin veneer on what is typically unprincipled imperative code. All the effect type polymorphism in the world can’t change this.</p>
<h2 id="recommendations">Recommendations</h2>
<p>In light of this analysis of tagless-final, I have some concrete recommendations for different situations:</p>
<ol>
<li><strong>Stay the Course</strong>. If you’re in a small, stable, high-skilled team that’s already using and benefiting from tagless-final due to a working social contract, then <em>keep using tagless-final</em>. Long-time functional programmers may “forget” how to program procedurally and instinctively confine themselves to a functional subset of Scala, which makes enforcement of the contract easier. Consider abandoning tagless-final (or building a compiler plug-in) if the team starts scaling.</li>
<li><strong>Build a Library</strong>. If you’re building an open source library that doesn’t take advantage of effect-specific features, then use an existing layer of indirection, such as Cats Effect. Or if you want to keep dependencies small, create a tiny layer of indirection for just the features you need. This layer may not help you reason about or test code, but it will help you support the full functional Scala market, which will improve adoption of your library.</li>
<li><strong>Ditch Tagless-Final</strong>. In all other cases, I recommend picking a concrete effect type (ZIO, Cats IO, or Monix). If you decide to switch later, you’ll pay a one-time cost for the (straightforward) refactor. Encourage coding to <em>interfaces</em>, not implementations. This social contract doesn’t scale either, but at least many Java developers are already indoctrinated in the practice. This will give you testability, it can be done incrementally, and it can give you the same (discipline-powered) reasoning benefits as tagless-final, if employed to the same extent.</li>
<li><strong>Minimize Effectful Code</strong>. Consider minimizing effectful code. Look for real abstractions in your domain, which are equipped with algebraic laws that help you reason generically about polymorphic code. Prefer declarative code instead of imperative (monadic) code. Don’t presume your <code class="language-plaintext highlighter-rouge">State</code> monad code is any better than its <code class="language-plaintext highlighter-rouge">IO</code> equivalent (it’s not). Prefer data types whose operations have denotational semantics.</li>
</ol>
<h1 id="summary">Summary</h1>
<p>Tagless-final has a brilliant sales pitch. It promises to future-proof our code to changes in concrete effect types. It promises us testability. It promises us the ability to reason about effects using parametric polymorphism.</p>
<p>Unfortunately, the reality of tagless-final doesn’t live up to the hype:</p>
<ul>
<li>Tagless-final does insulate us from an effect type, but that’s a maintenance burden and deprives us of useful principled operations and type-safety.</li>
<li>Tagless-final doesn’t provide us any testability, per se, and many common type classes prevent testability; it’s only coding to an interface that provides us with testability, which can be done with or without tagless-final.</li>
<li>Tagless-final doesn’t constrain effects, since Scala has no way to restrict side-effects; type signatures alone cannot tell us which side-effects are executed or modeled by a method.</li>
</ul>
<p>Beyond these drawbacks, real world tagless-final is littered with <em>sync bloat</em>, which can’t help us with unit testing or reasoning even if we are disciplined and diligent about restricting side-effects.</p>
<p>Further, since most tagless-final type classes are completely lawless, we can’t reason generically about code that uses them. True generic reasoning requires <em>abstractions</em>, defined by lawful sets of operations, and tagless-final doesn’t give us abstractions, only collections of ad hoc operations with unspecified semantics.</p>
<p>Tagless-final, far from being a pancea to managing functional effects, imposes great pedagogical and institutional costs. Many claim the technique renders Scala code bases impenetrable and unmaintainable. While an exaggeration, there is no question that the learning curve for tagless-final is steep, and the ergonomics of the technique are poor.</p>
<p>Ultimately, in my opinion, the benefits of tagless-final do not pay for the costs—at least not in most cases, and not with the current Scala compiler and tooling.</p>
<p>Small, stable teams that are already using tagless-final with a working social contract should probably keep using tagless-final, at least if they have found the benefits to outweigh the costs (and some teams have).</p>
<p>Developers of open source libraries can surely benefit from a layer of indirection around effect types, because even though indirection may not help with reasoning or testability, it can increase addressable market share.</p>
<p>Finally, other teams should probably avoid using tagless-final for managing effects, and embrace the age-old best practice of <em>coding to an interface</em>. While still just a social contract that relies on discipline, the practice is widely known, doesn’t require any fancy training, and can be used with more ergonomic approaches to testability and reasoning, including traditional dependency injection, module-oriented programming, and the reader monad.</p>
<p><a href="https://degoes.net/articles/tagless-horror">The False Hope of Managing Effects with Tagless-Final in Scala</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on June 18, 2019.</p>https://degoes.net/articles/zio-solo2019-05-28T00:00:00-06:002019-05-28T00:00:00-06:00John A De Goeshttps://degoes.netjohn@degoes.net<p>Many of my readers know I have promoted and contributed to <a href="https://github.com/scalaz/">Scalaz</a> for a little over two years now.</p>
<p>After many wonderful collaborations with some of the most talented Scala developers I know, I have decided to step back from Scalaz, and pull <a href="https://github.com/scalaz/scalaz-zio">ZIO</a>, my most significant work, into a separate organization—one that is not associated with Scalaz in any way.</p>
<p>I hope this post make it clear why I am doing this, and what it means for the future of functional programming in Scala.</p>
<h2 id="community-build-controversy">Community Build Controversy</h2>
<p>As most of the functional Scala community knows by now, all projects in the Scalaz organization, as well as some that depend on Scalaz, were <a href="https://github.com/scala/community-builds/commit/be8d3f07832c78044a65d3840a9338dc05943af7">recently removed from the Scala community build</a>.</p>
<p>The community build is used as a regression suite for the Scalac compiler, and helps ensure that changes to the compiler do not break innovative or widely-used libraries in the Scala ecosystem.</p>
<p>When I was made aware of this event, I <a href="https://github.com/scala/community-builds/issues/896">requested more information</a> and asked for a reversal.</p>
<p>When it became clear to me there would be no reversal or official explanation, I made a <a href="https://contributors.scala-lang.org/t/coc-compatible-community-builds/3097/240?u=jdegoes">final statement</a>, which expressed my disappointment and desire for better community stewardship, even while assuming good-faith all around.</p>
<p>Ultimately, some good came out of the discussions:</p>
<ul>
<li>The build is now more faithfully rebranded as the <a href="https://github.com/scala/community-builds/commit/df5a1c44f7e69f26737d09ffcefbc387c8bb1b97">Lightbend build</a>;</li>
<li>It is clear that which projects are included is entirely at the <a href="https://github.com/scala/community-builds/wiki/Eligibility/_compare/2d2b7e7e77f0d305452cd62fd5b27437eefc8571">discretion of Lightbend employees</a>, without input from the community.</li>
<li>There is a chance that ZIO (which did not depend on Scalaz, but lived inside the Scalaz organization), will eventually make it back into the Lightbend build, helping protect against compiler regression.</li>
</ul>
<p>While not the outcome I hoped for, these steps do represent progress toward more transparency, and I would hope only the Lightbend project be relocated to the <a href="https://github.com/lightbend">Lightbend organization</a> at some point in the future, to better convey the fact the build does not reflect what the Scala community uses or necessarily wishes to include in the build.</p>
<h2 id="the-birth-of-zio">The Birth of ZIO</h2>
<p>ZIO is a library for asynchronous and concurrent programming in Scala. While not dependent on Scalaz, the library has been hosted inside the Scalaz organization on Github, and has therefore been affected by the decision to remove all Scalaz projects from the Lightbend build.</p>
<p>ZIO was hosted inside Scalaz for historical reasons.</p>
<p>In early 2017, Vincent Marquez, a friend of mine and fellow functional Scala developer, asked me if I would be willing to contribute to the next version of Scalaz (Scalaz 8), which had a long tradition of pushing the boundaries of functional programming in Scala.</p>
<p>I said yes, and began looking for something interesting to work on.</p>
<p>For many years, I’ve been interested in how functional programming can improve asynchronous and concurrent programming. Having written several versions of <code class="language-plaintext highlighter-rouge">Future</code>, in several programming languages, and having authored the first version of <code class="language-plaintext highlighter-rouge">Aff</code> (the de facto asynchronous effect system for PureScript), working on the next-generation Scalaz effect system seemed like a good fit.</p>
<p>So I signed up to work on the <em>Scalaz 8 IO monad</em>.</p>
<p>I had heretical ideas about how such an effect type should be designed: I wanted built-in concurrency powered by fibers; I wanted resource-safety with iron-clad guarantees; I wanted fine-grained interruption (termed <em>async exceptions</em> in Haskell); and I wanted performance to radically improve on last-generation effect types.</p>
<p>While my ideas were extremely controversial at the time, I presented my work at Scala IO 2017 and Scale by the Bay 2017, and the reception was so overwhelmingly positive that it went on to permanently change the direction of the Cats IO data type and Cats Effect type class hierarchy.</p>
<p>Development of the Scalaz 8 effect system proceeded quickly, much faster than development of Scalaz 8 itself. Meanwhile, both Scalaz 7.x and Cats developers were interested in using the effect system even while still early in development. The demand was so high that other developers pulled the effect type out into a standalone project called <code class="language-plaintext highlighter-rouge">ioeffect</code>—which happened in about 24 hours and without any help from me.</p>
<p>When this happened, it became clear to me the best way to help the entire functional Scala community was to pull the effect system out of Scalaz 8, and make it a proper standalone project, without any dependencies on Scalaz or Cats, but with full support for both libraries in separate modules.</p>
<p>On June 11, 2018, <em>ZIO</em> was officially born.</p>
<h2 id="zio-drifts-from-scalaz">ZIO Drifts from Scalaz</h2>
<p>Over the coming months, ZIO got its own chat room on Gitter, its own set of contributors (most of whom did not use Scalaz, and some of whom used Cats!), and its own unique culture—which is overwhelmingly positive, supportive, and focused on mentoring and growing Scala developers.</p>
<p>ZIO was becoming its own thing, independent of Scalaz, even while it lived on within the same organization.</p>
<h2 id="the-scalaz-connection">The Scalaz Connection</h2>
<p>Having contributed to Scalaz, and worked (mostly on ZIO) within the Scalaz organization for more than two years, I can say unreservedly there are some amazing things about the Scalaz community.</p>
<p>Firstly, many of the <em>people</em> who work on Scalaz are very gifted, selfless individuals, who prize community service above all else. For example, Kenji Yoshida, Tomas Mikula, and others.</p>
<p>Secondly, there is a culture of technical excellence in Scalaz. Technical discussions are encouraged, even if they are critical. Contributors understand that <em>they are not their code</em>, and they welcome suggestions to improve their work.</p>
<p>Yet regardless of these cultural highlights, it is impossible to ignore the controversial history of the project. And at the center of this controversy is the project’s founder, <em>Tony Morris</em>.</p>
<p>Tony Morris is a long-time functional programmer, an accomplished teacher who has donated countless hours to developers, patiently teaching them everything from the elementary to the advanced (I remember him writing 50 lines of code to explain to me how <code class="language-plaintext highlighter-rouge">List</code> is a free monoid!).</p>
<p>Yet, while many (including myself) will attest to Tony’s generous assistance and mentorship over the years, there are others who have been offended or hurt by Tony’s communication style, or have found his relentless critiques of Scala to be emotionally draining.</p>
<p>I have witnessed Tony insult others on a number of occassions—not just critique their work, mind you, but engage in ad hominem. While, as far as I have seen, this was always in response to (more subtle and socially sanctioned) ad hominem, it does not mitigate the very real hurt caused by Tony’s actions.</p>
<p>The effects of this over time are twofold:</p>
<ol>
<li>Scalaz <em>proper</em> contributors are not easily offended and are much more direct (<em>selection bias</em>);</li>
<li>A few people refuse to interact with the Scalaz project because they have been hurt or are afraid of being hurt.</li>
</ol>
<p>Tony Morris is not the only person to scare developers away from Scala communities, of course. Some developers publicly refuse to contribute to Typelevel projects (and others, privately, so I am told), citing the actions of Lars Hupel or—before they “stepped down” from Typelevel—the actions of Travis Brown and Stew O’Connor, who still show up in the chat rooms and comment on project issues.</p>
<p>I’m not privy to everything, but as far as I can tell, Tony’s actions have been more publicly visible than those of Lars, Travis, or Stew, and have led to more public backlash.</p>
<h2 id="modern-day-scalaz">Modern Day Scalaz</h2>
<p>With the Scalaz mentorship program I led last year, the organization saw a huge influx of new contributors, who knew nothing about the history of the project or Tony’s communication style.</p>
<p>Most of the new contributors now work in the ZIO project, which is isolated from the main Scalaz project, but there are other Scalaz projects that work in their own silos (Parsers, Schema, etc.).</p>
<p>Although Tony Morris stopped contributing to Scalaz years ago (after he gave up on Scala), his ghost still haunts the halls of Scalaz to this day.</p>
<p>Tony is no longer present in the day-to-day, but he still shows up occassionally in the Scalaz chat room. Very rarely, he comments on issues in the Scalaz project. And, as he will likely do for many years to come, Tony continues to heavily criticize Scala on technical grounds.</p>
<p>For better or worse, the project founded by Tony Morris will probably always be associated with Tony Morris.</p>
<h2 id="zio--scalaz">ZIO != Scalaz</h2>
<p>I believe that every project and every community needs to be judged based on its own merits, and with the community build controversy, it has become clear to me this is not happening right now.</p>
<p>ZIO is not Scalaz, and I am not Tony Morris.</p>
<p>Many first-time contributors and users of ZIO have stated in public or in private that ZIO has one of the friendliest, most welcoming, and most supportive environments of <em>any</em> open source community.</p>
<p>The project has attracted more than a hundred diverse contributors from all around the world, some of whom chose ZIO to be their first contribution to the world of open source software.</p>
<p>I strongly believe that ad hominem will never help, but will make every situation worse. I believe that rather than trade barbs, leaders of a community should turn the other cheek (which may not be <em>fair</em> but it is <em>noble</em>), using empathy and nonviolent communication to heal and bind together.</p>
<p>It is not up to me to force my own personal communication philosophy on the open source projects of others. But it <em>is</em> up to me to follow my own path, and create the kind of communities I want to see in this world.</p>
<p>So in that spirit, and for both social as well as technical reasons, on April 29th I announced that ZIO would be leaving the Scalaz organization, and moving to a new <a href="https://github.com/zio">ZIO</a> organization on Github.</p>
<p>This is a brand new start for an exciting young project that will henceforth be judged on its own merits, for its own culture.</p>
<h2 id="life-after-scalaz">Life After Scalaz</h2>
<p>With ZIO moving outside of Scalaz, I hope to continue to build upon the culture we have created, mentoring new generations of functional Scala developers, and building an environment where everyone feels supported and empowered to become the best version of themselves.</p>
<p>I hope also to retain the many positive aspects of the Scalaz culture, such as a commitment to technical excellence, and a willingness to hear technical critiques and act on them without ego (easier said than done!).</p>
<p>I hope to continue the Scalaz tradition of questioning dogma.</p>
<p>For example, unlike many of my functional Scala comrades, I believe the key to making functional programming in Scala feel natural is to embrace all of Scala’s features, including subtyping, intersection types, and variance (without compromising purity); and to reject many Haskellisms, such as monad transformers, which have poor ergonomics and performance in Scala.</p>
<p>I believe that functional Scala programmers need to roll up their sleeves and help make Scala even better at functional programming, rather than criticize from the side-lines (something I have been guilty of myself!).</p>
<p>I do not anticipate contributing significantly to the current versions of Scalaz or Cats. Rather, I hope, and will encourage, a unification between Cats and Scalaz—a single base of contributors all collectively working toward making functional Scala wildly successful in industry.</p>
<p>As you might suspect, I have a few (heretical) ideas about what a unified architecture could look like, and I think it quite likely these ideas will materialize, in an appropriate way, and at a suitable time.</p>
<h2 id="closing-thoughts">Closing Thoughts</h2>
<p>I’m very grateful for the chance to participate in the development of the Scalaz library—a great library that first made functional programming in Scala viable.</p>
<p>I’m grateful that many of the relationships I’ve made in the Scalaz community will continue to live on, even as I step back from the project to focus on the new ZIO organization, and surrounding ecosystem.</p>
<p>I wish both Scalaz and Cats contributors the best of luck helping programmers derive value from functional programming. And I want to express my sincere hope that we find a way to bring together the whole functional Scala community, with a culture that’s positive, inclusive, supportive, and professional.</p>
<p>With work and a little luck, I believe that we can heal past hurt, mend trust, build bridges not walls, and end up doing something amazing that will be remembered for a generation of programmers.</p>
<p>See you all online at <a href="https://twitter.com/jdegoes">@jdegoes</a> or in the new <a href="https://gitter.im/zio/core">ZIO Chat Room</a>!</p>
<p><em><strong>Addendum</strong>: I want to make it clear that Tony Morris and I have been and remain friends, and that despite different goals and philosophies, I highly respect his expertise and contributions to functional programming, and I welcome his positive contributions to any conferences, events, or projects I’m involved with, subject to the same criteria as anyone else. I also acknowledge in writing this post that it is an incomplete picture, and if you’re interested in the full story of why Tony communicates in the way he does, you should talk to Tony.</em></p>
<p><a href="https://degoes.net/articles/zio-solo">Why I'm Stepping Back from Scalaz</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on May 28, 2019.</p>https://degoes.net/articles/when-to-typeclass2019-04-20T00:00:00-06:002019-04-20T00:00:00-06:00John A De Goeshttps://degoes.netjohn@degoes.net<p>Type classes are a powerful tool for abstraction in functional programming. But they are poorly understood, and often abused.</p>
<p>In this post, I want to explain what a type class is—in a way that will be <em>quite</em> foreign to most readers—and offer my own personal recommendations on when you should use type classes to solve a problem, and when you should avoid them.</p>
<p><em>Note: This post assumes you already know how to create and use type classes.</em></p>
<h2 id="type-class-101">Type Class 101</h2>
<p>In the simplest possible case, a <em>type class</em> is a function from a type to a set of lawful operations on values of that type.</p>
<p>Let’s take this definition apart, one piece at a time:</p>
<ol>
<li><em>A type class is a function …</em>. This is not a function in the <em>programming language</em> sense of the word, but rather, a function in the <em>mathematical</em> sense of the word. We give a type class some type, like <code class="language-plaintext highlighter-rouge">Int</code>, and we get back something (just <em>one</em> something). Moreover, when we give the type class the <em>same</em> type, we get back the <em>same</em> thing. How this function is encoded varies depending on the capabilities of the host language. For example, Scala and Haskell have quite different mechanisms for representing type classes.</li>
<li><em>…from a type…</em>. Technically, type classes can have multiple parameters, not just one; and technically, they may not be <em>types</em>, but rather, <em>type constructors</em>. But in the common case, type classes are functions that take one type. Some languages, like Java or Kotlin, don’t have higher-kinded type parameters.</li>
<li><em>…to a set…</em>. Like <em>function</em> above, this is not a <em>programming language</em> set, but rather, a <em>mathematical</em> set. Each operation in the set is unique (there are no duplicates). Like with “function” above, how the set is encoded varies depending on the programming language.</li>
<li><em>…of lawful operations…</em>. The operations provided by the type class are governed by laws. Laws are rather like unit tests, except they apply in <em>all</em> cases, not just <em>some</em> cases. A few programming languages (like Idris) can express laws in a way the compiler will enforce, but most programming languages rely on manual use of property-checking to verify laws are satisfied as part of a test suite.</li>
<li><em>…on values of that type</em>. The operations provided by the type class are all operations on <em>values</em> that have the type described in (2).</li>
</ol>
<p>A type class <em>instance</em>, on the other hand, is a <em>piecewise definition</em> of the type class function for a given type. The type class function is not defined for <em>all</em> types, but only for <em>some</em> types, and those definitions are described by type class <em>instances</em>. A type class is not defined for a given type when there does not exist an instance of the type class for that type. On the other hand, if a type class is defined for some type, then the instance provides the definition of the operations for that type.</p>
<p>Type classes compliment <em>parametric polymorphism</em>. Parametric polymorphism allows us to create data structures and functions that are <em>polymorphic</em> in a type <em>parameter</em>.</p>
<p>Parametric polymorphism serves two useful functions:</p>
<ol>
<li>It allows us to create reusable data structures and functions, which can be used across not just a single (monomorphic) type, but across lots of types;</li>
<li>It allows us to throw away information that is not relevant to a problem, constraining possible solutions, and improving guarantees of correctness.</li>
</ol>
<p>In the following code, we write <em>parametrically polymorphic</em> code that accesses the 2nd element of a tuple:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">snd</span><span class="o">[</span><span class="kt">A</span>, <span class="kt">B</span><span class="o">](</span><span class="n">tuple</span><span class="k">:</span> <span class="o">(</span><span class="kt">A</span><span class="o">,</span> <span class="kt">B</span><span class="o">))</span><span class="k">:</span> <span class="kt">B</span> <span class="o">=</span> <span class="nv">tuple</span><span class="o">.</span><span class="py">_2</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">snd</span> <span class="o">::</span> <span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span> <span class="o">-></span> <span class="n">b</span>
<span class="n">snd</span> <span class="p">(</span><span class="kr">_</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span> <span class="o">=</span> <span class="n">b</span></code></pre></figure>
<p>This code can work for all types <code class="language-plaintext highlighter-rouge">A</code> and <code class="language-plaintext highlighter-rouge">B</code>, and can therefore work for all tuples of arity 2. It is very resuable, and doesn’t know more about the types than it needs to.</p>
<p>While parametric polymorphism allows us to <em>throw away</em> structure, to make code more reusable and constraint implementations, sometimes it throws away <em>too much</em> structure to solve a given problem. In the above definition of <code class="language-plaintext highlighter-rouge">snd</code>, the function does not know <em>anything</em> about the types <code class="language-plaintext highlighter-rouge">A</code> and <code class="language-plaintext highlighter-rouge">B</code>, which is fine for this problem, but which is not fine for most problems.</p>
<p>Most polymorphic functions will require <em>some</em> structure from a type: they will need the ability to do <em>something</em> with values of that type. For example, they might need to combine them, compare them, iterate over them, or construct them.</p>
<p>To add structure to a type, to give us some <em>operations</em>, we use type classes. Type classes, being functions from types to a set of lawful operations on values of those types, give us the ability to operate with values of some unknown type. Although we may not know what the type is, thanks to type classes, we have enough structure to generically solve our problem.</p>
<p>The final piece of the puzzle is called a <em>type class constraint</em>. Type class constraints are the mechanism by which a language allows us to <em>statically require</em> that some polymorphic type have an <em>instance</em> for a given type class.</p>
<h3 id="example">Example</h3>
<p>In the following example, we define a type class called <code class="language-plaintext highlighter-rouge">Ord</code>, which provides a single lawful operation for some type <code class="language-plaintext highlighter-rouge">A</code>.</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Ordering</span>
<span class="k">object</span> <span class="nc">Ordering</span> <span class="o">{</span>
<span class="k">object</span> <span class="nc">LT</span> <span class="k">extends</span> <span class="nc">Ordering</span>
<span class="k">object</span> <span class="nc">EQ</span> <span class="k">extends</span> <span class="nc">Ordering</span>
<span class="k">object</span> <span class="nc">GT</span> <span class="k">extends</span> <span class="nc">Ordering</span>
<span class="o">}</span>
<span class="k">trait</span> <span class="nc">Ord</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">compare</span><span class="o">(</span><span class="n">l</span><span class="k">:</span> <span class="kt">A</span><span class="o">,</span> <span class="n">r</span><span class="k">:</span> <span class="kt">A</span><span class="o">)</span><span class="k">:</span> <span class="kt">Ordering</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">Ord</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">apply</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="k">implicit</span> <span class="n">A</span><span class="k">:</span> <span class="kt">Ord</span><span class="o">[</span><span class="kt">A</span><span class="o">])</span><span class="k">:</span> <span class="kt">Ord</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=</span> <span class="n">A</span>
<span class="o">}</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="kr">data</span> <span class="kt">Ordering</span> <span class="o">=</span> <span class="kt">LT</span> <span class="o">|</span> <span class="kt">EQ</span> <span class="o">|</span> <span class="kt">GT</span>
<span class="kr">class</span> <span class="kt">Ord</span> <span class="n">a</span> <span class="kr">where</span>
<span class="n">compare</span> <span class="o">::</span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span> <span class="o">-></span> <span class="kt">Ordering</span></code></pre></figure>
<p>In Scala, if we wish to obtain the operation for some type <code class="language-plaintext highlighter-rouge">A</code>, let’s say <code class="language-plaintext highlighter-rouge">Int</code>, we utilize the expression <code class="language-plaintext highlighter-rouge">Ord[Int]</code>, which gives us the <em>instance</em> for the type <code class="language-plaintext highlighter-rouge">Int</code>. If such an instance has not been defined, then we will get a compile-time error.</p>
<p>Similarly, in Scala, if we wish to add a <em>constraint</em> on some parametric parameter, we can use context bounds, which are how constraints are encoded in Scala:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">sort</span><span class="o">[</span><span class="kt">A:</span> <span class="kt">Ord</span><span class="o">](</span><span class="n">list</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">A</span><span class="o">])</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">A</span><span class="o">])</span></code></pre></figure>
<p>In Haskell, if we wish to obtain the operation for some type <code class="language-plaintext highlighter-rouge">A</code>, let’s say <code class="language-plaintext highlighter-rouge">String</code>, we cannot call the type class function directly, because instances are not first-class values in Haskell. However, we can use the type class operation, and Haskell will automatically “call” the type class function to retrieve the instance for that type. If such an instance has not been defined, we will get a compile-time error.</p>
<p>In Haskell, if we wish to add a constraint, we use a type class constraint:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="n">sort</span> <span class="o">::</span> <span class="nc">Ord</span> <span class="n">a</span> <span class="k">=></span> <span class="o">[</span><span class="kt">a</span><span class="o">]</span> <span class="o">-></span> <span class="o">[</span><span class="kt">a</span><span class="o">]</span> <span class="o">-></span> <span class="o">[</span><span class="kt">a</span><span class="o">]</span></code></pre></figure>
<p>In both of these examples, the use of parametric polymorphism allows us to throw away irrelevant details, making our sort function more generic, and constraining possible implementations, leading to stronger guarantees on correctness.</p>
<p>With this background out of the way, let’s talk about when to use type classes, and more importantly, when to <em>not</em> use them!</p>
<h2 id="type-class-best-practices">Type Class Best Practices</h2>
<p>The purpose of type classes is to add <em>structure</em> to polymorphic types—just enough structure to allow us to solve our problem, but not more structure than necessary, so we can benefit from maximum code reuse, and maximally constrain our implementation.</p>
<p>The word <em>structure</em> is critically important. By <em>structure</em>, I mean <em>algebraic structure</em>: the operations that a type class provides us with satisfy some properties (the <em>laws</em>) across <em>all types</em> for which there is an instance.</p>
<p>Without a notion of <em>structure</em>, we cannot know the meaning of generic code. In the previous example of <code class="language-plaintext highlighter-rouge">sort</code>, our polymorphic function requires the ability too totally order the elements of the list it is sorting. Without a notion of <em>total order</em>, embedded into the laws of the <code class="language-plaintext highlighter-rouge">Ord</code> type class, we cannot meaningfully define what it is to “sort” elements.</p>
<p>Structure allows <em>abstraction</em>: we can write code that is generic across a range of different types <em>precisely</em> because we can <em>define</em> the ways in which those types are similar. In the case of sorting, we can abstract over the element type, because we can define what it means to compare elements of a given type. Although <code class="language-plaintext highlighter-rouge">Int</code> and <code class="language-plaintext highlighter-rouge">String</code> are quite different, they both posess enough structure to provide a total ordering.</p>
<p>Abstraction is <em>not</em> the same as <em>indirection</em>. <em>Indirection</em> provides a layer of insulation between our code and concrete data types, and is useful for testability and modularity.</p>
<p>Indirection is not necessarily governed by laws (algebraic or otherwise). In object-oriented programming languages, interfaces are often used to provide <em>indirection</em>, where it is sometimes mistakenly called <em>abstraction</em>. In functional programming languages, records of functions or modules can be used to provide indirection.</p>
<p><em>The structure that type classes provide is the foundation for writing well-defined generic code; without structure, there is no abstraction—only indirection.</em></p>
<p>This leads to my rule of when to use type classes:</p>
<blockquote>
<p><strong>Consider using type classes whenever you can identify enough common structure across a range of data types that is by itself wholly sufficient for writing well-defined generic code across these data types</strong></p>
</blockquote>
<p>Conversely, if you cannot identify common structure (through a set of well-defined algebraic laws), then you cannot write well-defined generic code across different data types. This is a strong sign that what you really want is <em>indirection</em>, not abstraction.</p>
<p>This leads to my rule of when <strong>not</strong> to use type classes:</p>
<blockquote>
<p><strong>Consider using indirection whenever you cannot identify enough common structure across a range of data types that is wholly sufficient for writing well-defined generic code</strong></p>
</blockquote>
<p>Now sometimes we can define structure across a range of data types, but that structure is not wholly sufficient for writing generic code.</p>
<p>An example is the algebraic structure called <em>magma</em>. A <em>magma</em> provides an composition operation with <em>closure</em> (that i8s, if you compose two values of the same type, you get back another value of the same type). This is less structure than a semigroup, because semigroups also provide a guarantee of <em>associativity</em>.</p>
<p>The guarantee of closure is so weak, that we cannot write much (if any) generic code that only requires this guarantee. In fact, closure is so weak, no laws are necessary to express: a parametrically polymorphic type system provides us this guarantee “for free”.</p>
<p>This is why it is so critically important that the structure provided by the type class be sufficiently strong that useful generic code is possible.</p>
<p>If you are looking at some structure that is sufficiently rare so as to permit very few useful instances for a given type, then there’s a good chance that a type class will allow you to write generically useful code. Conversely, if you are looking at some structure that permits infinitely many equally useful instances, then there’s a good chance having a type class will not allow you to write generically useful code.</p>
<p>I consider these best practices for when to use and when not to use type classes. However, in the next section, I want to provide some practical reasons for following this advice, as well as show you what to do when type classes aren’t a good fit.</p>
<h2 id="the-power-of-values">The Power of Values</h2>
<p>Type classes provide a means of accessing structure as a function of type.</p>
<p>As I have defined them, anyway, a type class has a <em>single</em> instance for any type. If for any reason a type class had <em>multiple</em> instances for a given type, then this would imply that type classes are <em>not</em> functions—at least, not <em>functions</em> in the mathematical sense of the word, because they would map the same type to <em>different</em> sets of operations.</p>
<p>Such non-determinism would mean that any given type would no longer map to a unique set of operations. It would further imply that we could not know, just looking at a concrete type, what set of operations a type class would give us access to.</p>
<p>Indeed, one of the chief problems of introducing non-determinism into my definition of a type class is that simple refactorings, like moving a function from one file to another, can change the runtime behavior of your program—a fact that Scala developers know all too well, thanks to their exposure to <em>implicit abuse</em>.</p>
<p>Ensuring that type classes are <em>globally coherent</em> gives us the ability to clearly and easily understand how our program will behave, and to preserve this behavior in the presence of refactoring, without having to be cautious.</p>
<p>Yet, while global coherence (<em>determinism</em> of the type class function) is important for reasoning and refactoring, there is something distinctly anti-modular about type classes. Type classes are, after all, <em>global</em>, <em>piecewise</em> defined functions from types to sets of operations.</p>
<p><em>Global</em> definitions are opposed to <em>local</em> definitions, and <em>local, compositional definitions</em> are the foundation of <em>modularity</em>.</p>
<p>Herein lies the fundamental tradeoff of type classes: type classes provide global (anti-modular), type-directed reasoning that stems from the very algebraic structure of the types in question; while modules and their weaker variants (records of functions), provide local (modular), value-directed reasoning that stems from the makeup of the values themselves.</p>
<p>Although the battle between type classes and modules goes back for decades, there is no real competition, because while type classes are a natural fit for algebraic structure, modules (or records of functions) are a natural fit for indirection.</p>
<p>Indirection gives us loose coupling between pieces of the whole, allowing us to, for example, connect to different types of databases at runtime, accessing them through a uniform interface; allowing us to support local logging, but also remote logging; allowing us to have a local file system, and a remote one; and allowing us to have test implementations, as well as production implementations.</p>
<p>Type classes give us algebraic structure in the small, and indirection gives us modularity in the large. They do not compete with each other, but rather, compliment each other, and the weaknesses of the one are the strengths of the other.</p>
<p>Some beautiful properties of modules (and records of functions) include the following:</p>
<ul>
<li><strong>Superior Composition</strong>. We can take values, and compose them with other values, to yield values. While type class composition is possible, it looks and feels awkward because it has to be encoded in a different language than the primary, expression-oriented language that we write most of our code in.</li>
<li><strong>No Dependence on Type</strong>. While type classes are functions of types to values, modules are just ordinary values, so for a given type (or set of types), we can have as many different modules as we like. For example, we can have a different database module for each underlying database type (MySQL, PostgreSQL, SQL Server, etc.), not connected to the type of effect we are using or any other type.</li>
<li><strong>Superior Customization</strong>. We can easily and programmatically mix and match pieces of different modules, for example, drilling into a module and adding instrumentation to a single operation. Optics make these types of customizations trivial, but optics work on data, and type classes are not data.</li>
</ul>
<p>The benefits of working with ordinary values are tremendous and should not be overlooked. Now it’s also true that type classes have their own benefits. The lure of these benefits is sometimes so strong, it leads to <em>type class abuse</em>.</p>
<h2 id="type-class-abuse">Type Class Abuse</h2>
<p>Type class abuse is what happens when we use a type class to represent something that <em>should</em> be a module (or record of functions).</p>
<p>Typical symptoms of type class abuse include the following:</p>
<ul>
<li>The type class has <em>no laws</em> that permit well-defined generic use of the type class</li>
<li>There are <em>many</em> possible useful instances of the type class for a given type</li>
<li>The type class exists primarily to avoid passing or composing values</li>
</ul>
<p><a href="https://degoes.net/articles/when-to-typeclass">Type Classes: When To Use Them, When To Avoid Them</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on April 20, 2019.</p>https://degoes.net/articles/zio-cats-effect2019-04-18T00:00:00-06:002019-04-18T00:00:00-06:00John A De Goeshttps://degoes.netjohn@degoes.net<p>Cats Effect has become the <a href="https://www.reactive-streams.org">“Reactive Streams”</a> of the functional Scala world, enabling a diverse ecosystem of libraries to work together.</p>
<p>Many great libraries like http4s, FS2, and Doobie are built on the Cats Effect type classes, and effect libraries like <a href="https://github.com/scalaz/scalaz-zio">ZIO</a> and Monix provide instances of these type classes for their effect types.</p>
<p>Although not without a few drawbacks, many of which will be <a href="https://github.com/typelevel/cats-effect/issues/321">rectified in 3.0</a>, the Cats Effect library is helping many open source contributors economically support the whole functional Scala ecosystem.</p>
<p>Application developers who use Cats Effect face a far more difficult choice: which of the major effect types they will use to build their applications.</p>
<p>Application developers have three major choices:</p>
<ul>
<li>Cats IO, the reference implementation in Cats Effect</li>
<li>Monix, with its <code class="language-plaintext highlighter-rouge">Task</code> data type and associated reactive machinery</li>
<li>More recently, ZIO, with its <code class="language-plaintext highlighter-rouge">ZIO</code> data type and concurrent machinery</li>
</ul>
<p>In this post, I’m going to argue that if you are building a Cats Effect application, then ZIO provides a compelling choice, one with design choices and features quite different than the Cats IO reference implementation.</p>
<p>Without further ado, let’s take a look at my top 12 reasons why ZIO and Cats Effect are a match made in heaven!</p>
<h1 id="1-better-mtl--tagless-final">1. Better MTL / Tagless-Final</h1>
<p>MTL, which stands for <em>Monad Transformers Library</em>, is a style of programming where functions are <em>polymorphic</em> in their effect type, expressing their requirements through <em>type class constraints</em>.</p>
<p>In Scala, this is often called the <em>tagless-final</em> style (although they are not exactly the same thing), especially when the type classes have no laws.</p>
<p>It is well-known that it is impossible to define global instances for such classic MTL type classes as <em>Writer</em> and <em>State</em> for effect types like Cats IO.</p>
<p>The reason is that the instances of these type classes for effect types requires access to mutable state, which cannot be created globally, because the creation of mutable state is effectful.</p>
<p>For <a href="/articles/effects-without-transformers">performance reasons</a>, however, it’s critical to avoid monad transformers, and provide an implementation of <em>Writer</em> and <em>State</em> directly atop the underlying effect type.</p>
<p>To accomplish this, functional Scala developers use a trick: they effectfully (but purely) create instances at the top level of their program, and then provide them downstream as local implicits:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="nv">Ref</span><span class="o">.</span><span class="py">make</span><span class="o">[</span><span class="kt">AppState</span><span class="o">](</span><span class="n">initialAppState</span><span class="o">).</span><span class="py">flatMap</span><span class="o">(</span><span class="n">ref</span> <span class="k">=></span>
<span class="k">implicit</span> <span class="k">val</span> <span class="nv">monadState</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">MonadState</span><span class="o">[</span><span class="kt">Task</span>, <span class="kt">AppState</span><span class="o">]</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">get</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">AppState</span><span class="o">]</span> <span class="k">=</span> <span class="nv">ref</span><span class="o">.</span><span class="py">get</span>
<span class="k">def</span> <span class="nf">set</span><span class="o">(</span><span class="n">s</span><span class="k">:</span> <span class="kt">AppState</span><span class="o">)</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span> <span class="nv">ref</span><span class="o">.</span><span class="py">set</span><span class="o">(</span><span class="n">s</span><span class="o">).</span><span class="py">unit</span>
<span class="o">}</span>
<span class="n">myProgram</span>
<span class="o">)</span></code></pre></figure>
<p>Although this trick is useful, it is also a hack. In a perfect world, all type class instances would be globally coherent (<em>one instance per type</em>)—not created locally and effectfully and then magically turned into implicit values to feed downstream methods.</p>
<p>A remarkable property about MTL / tagless-final is that you can <em>directly</em> define most instances atop the ZIO data type, by using <a href="/articles/zio-environment">ZIO Environment</a>.</p>
<p>Here’s one way to create a global definition of <code class="language-plaintext highlighter-rouge">MonadState</code> for the ZIO data type:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">trait</span> <span class="nc">State</span><span class="o">[</span><span class="kt">S</span><span class="o">]</span>
<span class="nc">def</span> <span class="n">state</span><span class="k">:</span> <span class="kt">Ref</span><span class="o">[</span><span class="kt">S</span><span class="o">]</span>
<span class="o">}</span>
<span class="k">implicit</span> <span class="k">def</span> <span class="nf">ZIOMonadState</span><span class="o">[</span><span class="kt">S</span>, <span class="kt">R</span> <span class="k"><:</span> <span class="kt">State</span><span class="o">[</span><span class="kt">S</span><span class="o">]</span>, <span class="kt">E</span><span class="o">]</span><span class="k">:</span> <span class="kt">MonadState</span><span class="o">[</span><span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">?</span><span class="o">]</span>, <span class="kt">S</span><span class="o">]</span> <span class="k">=</span>
<span class="k">new</span> <span class="nc">MonadState</span><span class="o">[</span><span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">?</span><span class="o">]</span>, <span class="kt">S</span><span class="o">]</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">get</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">S</span><span class="o">]</span> <span class="k">=</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">accessM</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">state</span><span class="o">.</span><span class="py">get</span><span class="o">)</span>
<span class="k">def</span> <span class="nf">set</span><span class="o">(</span><span class="n">s</span><span class="k">:</span> <span class="kt">S</span><span class="o">)</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">accessM</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">state</span><span class="o">.</span><span class="py">set</span><span class="o">(</span><span class="n">s</span><span class="o">).</span><span class="py">unit</span><span class="o">)</span>
<span class="o">}</span></code></pre></figure>
<p>This instance is now defined globally for any environment that supports at least <code class="language-plaintext highlighter-rouge">State[S]</code>.</p>
<p>Similarly for <code class="language-plaintext highlighter-rouge">FunctorListen</code>, otherwise known as <code class="language-plaintext highlighter-rouge">MonadWriter</code>:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">trait</span> <span class="nc">Writer</span><span class="o">[</span><span class="kt">W</span><span class="o">]</span>
<span class="nc">def</span> <span class="n">writer</span><span class="k">:</span> <span class="kt">Ref</span><span class="o">[</span><span class="kt">W</span><span class="o">]</span>
<span class="o">}</span>
<span class="k">implicit</span> <span class="k">def</span> <span class="nf">ZIOFunctorListen</span><span class="o">[</span><span class="kt">W:</span> <span class="kt">Semigroup</span>, <span class="kt">R</span> <span class="k"><:</span> <span class="kt">Writer</span><span class="o">[</span><span class="kt">W</span><span class="o">]</span>, <span class="kt">E</span><span class="o">]</span><span class="k">:</span> <span class="kt">FunctorListen</span><span class="o">[</span><span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">?</span><span class="o">]</span>, <span class="kt">W</span><span class="o">]</span> <span class="k">=</span>
<span class="k">new</span> <span class="nc">FunctorListen</span><span class="o">[</span><span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">?</span><span class="o">]</span>, <span class="kt">W</span><span class="o">]</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">listen</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">fa</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">A</span><span class="o">])</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="o">(</span><span class="kt">A</span>, <span class="kt">W</span><span class="o">)]</span> <span class="k">=</span>
<span class="nv">ZIO</span><span class="o">.</span><span class="py">accessM</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">state</span><span class="o">.</span><span class="py">get</span><span class="o">.</span><span class="py">flatMap</span><span class="o">(</span><span class="n">w</span> <span class="k">=></span> <span class="nv">fa</span><span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="n">a</span> <span class="k">=></span> <span class="n">a</span> <span class="o">-></span> <span class="n">w</span><span class="o">)))</span>
<span class="k">def</span> <span class="nf">tell</span><span class="o">(</span><span class="n">w</span><span class="k">:</span> <span class="kt">W</span><span class="o">)</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">W</span><span class="o">]</span> <span class="k">=</span>
<span class="nv">ZIO</span><span class="o">.</span><span class="py">accessM</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">state</span><span class="o">.</span><span class="py">update</span><span class="o">(</span><span class="k">_</span> <span class="o">|+|</span> <span class="n">w</span><span class="o">).</span><span class="py">unit</span><span class="o">)</span>
<span class="o">}</span></code></pre></figure>
<p>And of course, we can do the same for <code class="language-plaintext highlighter-rouge">MonadError</code>:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">implicit</span> <span class="k">def</span> <span class="nf">ZIOMonadError</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span><span class="o">]</span><span class="k">:</span> <span class="kt">MonadError</span><span class="o">[</span><span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">?</span><span class="o">]</span>, <span class="kt">E</span><span class="o">]</span> <span class="k">=</span>
<span class="k">new</span> <span class="nc">MonadError</span><span class="o">[</span><span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">?</span><span class="o">]</span>, <span class="kt">E</span><span class="o">]{</span>
<span class="k">def</span> <span class="nf">handleErrorWith</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">fa</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">A</span><span class="o">])(</span><span class="n">f</span><span class="k">:</span> <span class="kt">E</span> <span class="o">=></span> <span class="nc">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">A</span><span class="o">])</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">A</span><span class="o">]</span> <span class="k">=</span>
<span class="n">fa</span> <span class="n">catchAll</span> <span class="n">f</span>
<span class="k">def</span> <span class="nf">raiseError</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">e</span><span class="k">:</span> <span class="kt">E</span><span class="o">)</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">A</span><span class="o">]</span> <span class="k">=</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">fail</span><span class="o">(</span><span class="n">e</span><span class="o">)</span>
<span class="o">}</span></code></pre></figure>
<p>This technique is readily applicable to other type classes, including tagless-final type classes, whose instances may require effectfully-created state (mutable, configuration, etc.), testable effectful functions (combining environmental effects with tagless-final), or anything else that is readily accessible from the environment.</p>
<p>So if you love using MTL-style, or find the benefits of tagless-final outweigh the costs, then using ZIO lets you easily define <em>global</em> instances for all your favorite type classes.</p>
<p>No slow monad transformers, no effectfully created type class instances, no local implicits, and no hacks. Just straight up pure functional programming!</p>
<h1 id="2-resource-safety-for-mortals">2. Resource-Safety for Mortals</h1>
<p>An early defining feature of ZIO was <em>interruption</em>, the ability for the ZIO runtime to instantaneously cancel any executing effect, safely cleaning up all resources; and a course-grained version of this feature eventually made its way into Cats IO.</p>
<p>This feature, called <em>async exceptions</em> in Haskell, allows composable and efficient timeouts, efficient parallel and race operations, and globally efficient computation.</p>
<p>While extremely powerful, interruption poses unique challenges for resource safety.</p>
<p>Programmers are mostly used to mentally tracking failure in their applications, or ZIO uses the type system to help track failure. But interruption is different. An effect composed from many other effects can be interrupted at <em>any</em> boundary.</p>
<p>Take the following effect:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">for</span> <span class="o">{</span>
<span class="n">handle</span> <span class="k"><-</span> <span class="nf">openFile</span><span class="o">(</span><span class="n">file</span><span class="o">)</span>
<span class="n">data</span> <span class="k"><-</span> <span class="nf">readFile</span><span class="o">(</span><span class="n">handle</span><span class="o">)</span>
<span class="k">_</span> <span class="k"><-</span> <span class="nf">closeFile</span><span class="o">(</span><span class="n">handle</span><span class="o">)</span>
<span class="o">}</span> <span class="k">yield</span> <span class="n">data</span></code></pre></figure>
<p>Most programmers would not be surprised that if <code class="language-plaintext highlighter-rouge">readFile</code> failed, then the <code class="language-plaintext highlighter-rouge">closeFile</code> would not be executed. Fortunately, effect systems have <code class="language-plaintext highlighter-rouge">ensuring</code> (called <code class="language-plaintext highlighter-rouge">guarantee</code> in Cats Effect) that lets you add a finalizer to an effect, similar to <code class="language-plaintext highlighter-rouge">finally</code>.</p>
<p>So the main problem with the above effect can be fixed simply:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">for</span> <span class="o">{</span>
<span class="n">handle</span> <span class="k"><-</span> <span class="nf">openFile</span><span class="o">(</span><span class="n">file</span><span class="o">)</span>
<span class="n">data</span> <span class="k"><-</span> <span class="nf">readFile</span><span class="o">(</span><span class="n">handle</span><span class="o">).</span><span class="py">ensuring</span><span class="o">(</span><span class="nf">closeFile</span><span class="o">(</span><span class="n">handle</span><span class="o">))</span>
<span class="o">}</span> <span class="nf">yield</span> <span class="o">()</span></code></pre></figure>
<p>Now the effect is <em>failure-proof</em>, in the sense that if <code class="language-plaintext highlighter-rouge">readFile</code> fails, then the file will be closed; and if <code class="language-plaintext highlighter-rouge">readFile</code> succeeds, the file will be closed; so in “all” cases, the file will be closed.</p>
<p>Well, not quite <em>all</em>. <em>Interruption</em> means that the executing effect can be terminated anywhere, even <em>between</em> the <code class="language-plaintext highlighter-rouge">openFile</code> and the <code class="language-plaintext highlighter-rouge">readFile</code>. If this happens, then the opened resource will not be closed, and a leak will result.</p>
<p>This pattern of acquiring and releasing a resource is so common, that ZIO introduced a <code class="language-plaintext highlighter-rouge">bracket</code> operator that made its way to Cats Effect 1.0.</p>
<p>The <code class="language-plaintext highlighter-rouge">bracket</code> operator is <em>interruption-proof</em>: if the acquire succeeds, then release will be called, no matter what, even if the effect that uses the resource is interrupted. Further, neither the acquire nor release can be interrupted, providing a strong guarantee of resource safety.</p>
<p>With <code class="language-plaintext highlighter-rouge">bracket</code>, the above example looks like this:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="nf">openFile</span><span class="o">(</span><span class="n">file</span><span class="o">).</span><span class="py">bracket</span><span class="o">(</span><span class="nf">closeFile</span><span class="o">(</span><span class="k">_</span><span class="o">))(</span><span class="nf">readFile</span><span class="o">(</span><span class="k">_</span><span class="o">))</span></code></pre></figure>
<p>Unfortunately, <code class="language-plaintext highlighter-rouge">bracket</code> only encapsulates one (particularly common) pattern of resource consumption; there are many others, especially with concurrent data structures, whose acquisition must be <em>interruptible</em> in order to avoid a different kind of leak.</p>
<p>In general, when programming with interruption, there are two things we want to do:</p>
<ul>
<li>Prevent interruption from happening in some region that is otherwise interruptible</li>
<li>Allow interruption to happen in some region that is otherwise uninterruptible</li>
</ul>
<p>ZIO has facilities to make both of these very easy. For example, we can implement our own version of <code class="language-plaintext highlighter-rouge">bracket</code> using lower-level features built into ZIO:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="nv">ZIO</span><span class="o">.</span><span class="py">uninterruptible</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">a</span> <span class="k"><-</span> <span class="n">acquire</span>
<span class="n">exit</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">interruptible</span><span class="o">(</span><span class="nf">use</span><span class="o">(</span><span class="n">a</span><span class="o">)).</span><span class="py">run</span><span class="o">.</span><span class="py">flatMap</span><span class="o">(</span><span class="n">exit</span> <span class="k">=></span> <span class="nf">release</span><span class="o">(</span><span class="n">a</span><span class="o">,</span> <span class="n">exit</span><span class="o">).</span><span class="py">const</span><span class="o">(</span><span class="n">exit</span><span class="o">))</span>
<span class="n">b</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">done</span><span class="o">(</span><span class="n">exit</span><span class="o">)</span>
<span class="o">}</span> <span class="k">yield</span> <span class="n">b</span>
<span class="o">}</span></code></pre></figure>
<p>In this code, <code class="language-plaintext highlighter-rouge">use(a)</code> is the only part that can be interrupted, and the surrounding code takes care to execute <code class="language-plaintext highlighter-rouge">release</code> in any case.</p>
<p>Interruptibility can be arbitrarily checked, turned off, or turned on, and only <em>two</em> primitive operations are necessary (all others are derived from these).</p>
<p>This compositional, full-featured model of interruptibility allows not just a clean implementation of <code class="language-plaintext highlighter-rouge">bracket</code>, but clean implementations of other scenarios in resource handling, which carefully balance the tradeoffs inherit in interruptibility.</p>
<p>Cats IO chose to provide only a <em>single</em> operation to manage interruptibility: a combinator called <code class="language-plaintext highlighter-rouge">uncancelable</code>. This makes a whole region uninterruptible. However, by itself the operation is of limited use, and can easily lead to code that wastes resources or deadlocks.</p>
<p>While it turns out that one can define an operator that provides more control over interruption atop Cats IO, the (quite clever!) <a href="https://github.com/SystemFw/playground/blob/d84aebb5fc1d2ccc4328afdca7ec8e923ef5a288/src/main/scala/Playground.scala">implementation by Fabio Labella</a> is insanely complex and not performant.</p>
<p>ZIO lets anyone write interruption-friendly code, operating at a high-level, with declarative, composable operators, and doesn’t force you to either choose between extreme complexity and poor performance on the one hand, and wasted resources and deadlocks on the other.</p>
<p>Moreover, although not discussed in this post, the newly-added <a href="https://www.youtube.com/watch?list=PL8NC5lCgGs6MYG0hR_ZOhQLvtoyThURka&v=d6WWmia0BPM">Software Transactional Memory</a> in ZIO lets users declaratively write data structures and code that is automatically asynchronous, concurrent, and safely interruptible.</p>
<h1 id="3-guaranteed-finalizers">3. Guaranteed Finalizers</h1>
<p>The <code class="language-plaintext highlighter-rouge">try</code> / <code class="language-plaintext highlighter-rouge">finally</code> construct in many programming languages provides us the robust guarantees we need to write synchronous code that doesn’t leak resources.</p>
<p>In particular, the construct provides the following guarantee:</p>
<ul>
<li>If the <code class="language-plaintext highlighter-rouge">try</code> block begins execution, then the <code class="language-plaintext highlighter-rouge">finally</code> block will begin execution when the <code class="language-plaintext highlighter-rouge">try</code> block stops execution</li>
</ul>
<p>This guarantee holds even if:</p>
<ul>
<li>There are nested <code class="language-plaintext highlighter-rouge">try</code> / <code class="language-plaintext highlighter-rouge">finally</code> blocks</li>
<li>There are errors in the <code class="language-plaintext highlighter-rouge">try</code> block</li>
<li>There are errors in a nested <code class="language-plaintext highlighter-rouge">finally</code> block</li>
</ul>
<p>ZIO’s <code class="language-plaintext highlighter-rouge">ensuring</code> operation can be used exactly like <code class="language-plaintext highlighter-rouge">try</code> / <code class="language-plaintext highlighter-rouge">finally</code>:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">effect2</span> <span class="k">=</span> <span class="nv">effect</span><span class="o">.</span><span class="py">ensuring</span><span class="o">(</span><span class="n">cleanup</span><span class="o">)</span></code></pre></figure>
<p>ZIO provides the following guarantee on <code class="language-plaintext highlighter-rouge">effect.ensuring(finalizer)</code>:</p>
<ul>
<li>If <code class="language-plaintext highlighter-rouge">effect</code> begins execution, then <code class="language-plaintext highlighter-rouge">finalizer</code> will begin execution when the <code class="language-plaintext highlighter-rouge">effect</code> stops execution</li>
</ul>
<p>Like <code class="language-plaintext highlighter-rouge">try</code> / <code class="language-plaintext highlighter-rouge">finally</code>, this guarantee holds even if:</p>
<ul>
<li>There are nested <code class="language-plaintext highlighter-rouge">ensuring</code> compositions</li>
<li>There are errors in <code class="language-plaintext highlighter-rouge">effect</code></li>
<li>There are errors in any nested finalizer</li>
</ul>
<p>Moreover, the guarantee holds even if the effect is <em>interrupted</em> (the guarantees on <code class="language-plaintext highlighter-rouge">bracket</code> are similar, and in fact, <code class="language-plaintext highlighter-rouge">bracket</code> is implemented on <code class="language-plaintext highlighter-rouge">ensuring</code>).</p>
<p>The Cats IO data type chose a different, weaker guarantee. For <code class="language-plaintext highlighter-rouge">effect.guarantee(finalizer)</code>, the guarantee is weakened as follows:</p>
<ul>
<li>If <code class="language-plaintext highlighter-rouge">effect</code> begins execution, then <code class="language-plaintext highlighter-rouge">finalizer</code> will begin execution when the <code class="language-plaintext highlighter-rouge">effect</code> stops execution, <strong>unless</strong> problematic effects are composed into <code class="language-plaintext highlighter-rouge">effect</code></li>
</ul>
<p>This weakening also occurs for the Cats IO implementation of <code class="language-plaintext highlighter-rouge">bracket</code>.</p>
<p>In order to leak resources, it is only necessary to compose, somewhere in the effect of <code class="language-plaintext highlighter-rouge">guarantee</code>, or inside the “use” effect of <code class="language-plaintext highlighter-rouge">bracket</code>, an effect similar to the following:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="c1">// Assume `interruptedFiber` is some fiber that is already interrupted:</span>
<span class="k">val</span> <span class="nv">bigTrouble</span> <span class="k">=</span> <span class="nv">interruptedFiber</span><span class="o">.</span><span class="py">join</span></code></pre></figure>
<p>When <code class="language-plaintext highlighter-rouge">bigTrouble</code> is so composed into another effect, the effect becomes <em>non-terminating</em>—neither finalizers installed with <code class="language-plaintext highlighter-rouge">guarantee</code> nor cleanup effects installed with <code class="language-plaintext highlighter-rouge">bracket</code> will be executed, leading to <em>resource leaks</em> and <em>skipped</em> finalization.</p>
<p>For example, the finalizer in the following code will never begin execution:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="o">(</span><span class="nv">IO</span><span class="o">.</span><span class="py">unit</span> <span class="o">>></span> <span class="n">bigTrouble</span><span class="o">).</span><span class="py">guarantee</span><span class="o">(</span><span class="nc">IO</span><span class="o">(</span><span class="nf">println</span><span class="o">(</span><span class="s">"Won't be executed!!!"</span><span class="o">)))</span></code></pre></figure>
<p>Using local reasoning, it is not possible to know if an effect like <code class="language-plaintext highlighter-rouge">bigTrouble</code> is being composed somewhere in the “use” effect of bracket or inside of a finalizer.</p>
<p>Therefore, you cannot know if a Cats IO program will leak resources or skip finalization without global program analysis. Global program analysis is a manual, error-prone process that cannot be checked by the compiler, and which must be repeated every time any relevant part of the code changes.</p>
<p>ZIO has custom implementations of the Cats Effect <code class="language-plaintext highlighter-rouge">guarantee</code>, <code class="language-plaintext highlighter-rouge">guaranteeCase</code>, and <code class="language-plaintext highlighter-rouge">bracket</code> operations. The implementations use native ZIO semantics (not Cats IO semantics), which allow you to reason locally about resource safety, knowing that in all cases, finalizers <em>will</em> be run and resources <em>will</em> be freed.</p>
<h1 id="4-stable-shifting">4. Stable Shifting</h1>
<p>Cats Effect has an <code class="language-plaintext highlighter-rouge">evalOn</code> method of <code class="language-plaintext highlighter-rouge">ContextShift</code>, which allows moving the execution of some code to another execution context.</p>
<p>This turns out to be quite handy for a number of reasons:</p>
<ul>
<li>Many client libraries require you to do some work in their thread pool</li>
<li>UI libraries require some updates to be done on the UI thread</li>
<li>Some effects need to be isolated on thread pools tailored for their specific needs</li>
</ul>
<p>The <code class="language-plaintext highlighter-rouge">evalOn</code> operation is designed to execute an effect where it needs to be run, and then hop back to the original execution context. For example:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="nv">cs</span><span class="o">.</span><span class="py">evalOn</span><span class="o">(</span><span class="n">kafkaContext</span><span class="o">)(</span><span class="n">kafkaEffect</span><span class="o">)</span></code></pre></figure>
<p><em>Note</em>: Cats IO has a related construct called <code class="language-plaintext highlighter-rouge">shift</code> that allows you to “hop” over to another context without hopping back, but in practice, this behavior is almost never desired, so the <code class="language-plaintext highlighter-rouge">evalOn</code> variation is strongly preferred.</p>
<p>ZIO’s implementation of <code class="language-plaintext highlighter-rouge">evalOn</code> (built on the ZIO primitive <code class="language-plaintext highlighter-rouge">lock</code>) provides a guarantee necessary for local reasoning about where effects are running:</p>
<ul>
<li>The effect will always execute on the specified context</li>
</ul>
<p>Cats IO chose a different, weaker guarantee:</p>
<ul>
<li>The effect will execute on the specified context <strong>until</strong> the first asynchronous operation or embedded shift</li>
</ul>
<p>Using just local reasoning, it is not possible to know if an asynchronous effect (or nested shift) is being composed into the effect being shifted, because asynchronicity is not reflected in types.</p>
<p>Therefore, as with resource safety, knowing where a Cats IO effect will run requires global program analysis. In practice, and from my experience, users of Cats IO are quite surprised when they use <code class="language-plaintext highlighter-rouge">evalOn</code> with one context, and then later find out that most of the effect has been accidentally executed on some other context.</p>
<p>ZIO lets you specify where effects should run and trust that will actually happen, in all cases, regardless of how effects are composed with other effects.</p>
<h1 id="5-lossless-errors">5. Lossless Errors</h1>
<p>Any effect type that supports concurrency, parallelism, or resource safety runs into an immediate problem with a linear error model: in general, errors don’t compose.</p>
<p>This holds both for <code class="language-plaintext highlighter-rouge">Throwable</code>, the fixed error type baked into Cats IO, and for polymorphic error types, which are supported by ZIO.</p>
<p>All the following situations can lead to multiple errors being produced:</p>
<ul>
<li>A finalizer throwing an exception</li>
<li>Two (failing) effects being combined in parallel</li>
<li>Two (failing) effects being raced</li>
<li>An interrupted effect also failing before exiting an uninterruptible section</li>
</ul>
<p>Because errors do not compose, ZIO has a data structure called <code class="language-plaintext highlighter-rouge">Cause[E]</code>, which provides a free <em>semiring</em> (an abstraction from abstract algebra, which you can safely ignore if you haven’t heard about before!), which allows lossless composition of sequential and parallel errors for any arbitrary error type.</p>
<p>During all operations (including cleanup for a failed or interrupted effect), ZIO aggregates errors into the <code class="language-plaintext highlighter-rouge">Cause[E]</code> data structure, which can be accessed at any time.</p>
<p>As a result, ZIO never loses any errors: they can all be accessed at the value level, and then logged, inspected, or transformed, as dictated by business requirements.</p>
<p>Cats IO chose to embrace a lossy error model. Wherever ZIO would compose two errors using <code class="language-plaintext highlighter-rouge">Cause[E]</code>, Cats IO “throws” one error away—for example, by calling <code class="language-plaintext highlighter-rouge">e.printStackTrace()</code> on the tossed error.</p>
<p>For example, the finalizer error in this snippet will be “thrown away”:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="nv">IO</span><span class="o">.</span><span class="py">raiseError</span><span class="o">(</span><span class="k">new</span> <span class="nc">Error</span><span class="o">(</span><span class="s">"Error 1"</span><span class="o">)).</span><span class="py">guarantee</span><span class="o">(</span><span class="nv">IO</span><span class="o">.</span><span class="py">raiseError</span><span class="o">(</span><span class="k">new</span> <span class="nc">Error</span><span class="o">(</span><span class="s">"Error 2"</span><span class="o">)))</span></code></pre></figure>
<p>This lossy side-channel error reporting means there is no way to locally detect and respond to the full range of errors that can occur as effects are composed.</p>
<p>ZIO lets you use any error type you want, including <code class="language-plaintext highlighter-rouge">Throwable</code> (or more specific subtypes of <code class="language-plaintext highlighter-rouge">Throwable</code>, like <code class="language-plaintext highlighter-rouge">IOException</code> or a custom exception hierarchy), giving you the guarantee that no errors will be lost during composition.</p>
<h1 id="6-deadlock-free-async">6. Deadlock-Free Async</h1>
<p>Both ZIO and Cats IO provide a constructor that allows one to take callback-based code, and lift it into an effect value.</p>
<p>This capability is exposed via the <code class="language-plaintext highlighter-rouge">Async</code> type class in Cats Effect:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">effect</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">Data</span><span class="o">]</span> <span class="k">=</span>
<span class="nc">Async</span><span class="o">[</span><span class="kt">Task</span><span class="o">].</span><span class="py">async</span><span class="o">(</span><span class="n">k</span> <span class="k">=></span>
<span class="nf">getDataWithCallbacks</span><span class="o">(</span>
<span class="n">onSuccess</span> <span class="k">=</span> <span class="n">v</span> <span class="k">=></span> <span class="nf">k</span><span class="o">(</span><span class="nc">Right</span><span class="o">(</span><span class="n">v</span><span class="o">)),</span>
<span class="n">onFailure</span> <span class="k">=</span> <span class="n">e</span> <span class="k">=></span> <span class="nf">k</span><span class="o">(</span><span class="nc">Left</span><span class="o">(</span><span class="n">e</span><span class="o">))</span>
<span class="o">))</span></code></pre></figure>
<p>This creates an asynchronous effect that, when executed, will suspend until the value is available, and then resume—all transparently to the user of the effect. This property is what makes functional effect systems so pleasing for asynchronous code.</p>
<p>Notice that as the callback-code is being lifted into the effect, a callback function (here called <code class="language-plaintext highlighter-rouge">k</code>) is invoked. This callback function is provided with the success or error value.</p>
<p>When this callback function is invoked, execution of the (suspended) effect resumes.</p>
<p>ZIO provides the guarantee the effect will resume executing on either the runtime’s default thread pool, if the effect has not been locked to a specific context, or on the specific context the effect has been locked to.</p>
<p>Cats IO chose to resume executing the effect on the thread invoking the callback.</p>
<p>The difference between these decisions is quite profound. In general, the thread that is invoking the callback does not expect the callback code to continue indefinitely; it expects a short delay before control is returned to the caller.</p>
<p>ZIO provides the guarantee that control is returned to the caller immediately, which can then resume execution normally.</p>
<p>On the other hand, Cats IO provides no such guarantee, which means the caller thread invoking the callback may get “stuck” waiting indefinitely for control to be returned to it.</p>
<p>Early versions of Cats Effect concurrent data structures (<code class="language-plaintext highlighter-rouge">Deferred</code>, <code class="language-plaintext highlighter-rouge">Semaphore</code>, etc.) resumed effects that did not promptly yield control back to the caller thread. As a result, they had problems with deadlocks and unfair scheduling. While all of these problems have been identified and fixed, they have only been fixed for <em>Cats Effect</em> concurrent data structures.</p>
<p>User-land code that uses a similar pattern with Cats IO will run into similar issues, and because of the nondeterminism involved, they may manfiest only occassionally, at runtime, making diagnosing and solving the issues challenging.</p>
<p>ZIO’s model provides deadlock safety and fairness by default, and forces users to opt into the Cats IO behavior explicitly (by, for example, using <code class="language-plaintext highlighter-rouge">unsafeRun</code> on a <code class="language-plaintext highlighter-rouge">Promise</code> that is completed from the resumed asynchronous effect).</p>
<p>While neither choice is suitable in all cases, and while both ZIO and Cats IO provide enough flexibility to handle all cases (in different ways), the ZIO choice means worry-free use of <code class="language-plaintext highlighter-rouge">Async</code>, and pushes problematic code to <code class="language-plaintext highlighter-rouge">unsafeRun</code>, which is already a known deadlock-risk.</p>
<h1 id="7-precise-future-interop">7. Precise Future Interop</h1>
<p>Dealing with Scala’s <code class="language-plaintext highlighter-rouge">Future</code> is a reality for many code bases. ZIO ships with a <code class="language-plaintext highlighter-rouge">fromFuture</code> method that provides a ready-made execution context:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="nv">ZIO</span><span class="o">.</span><span class="py">fromFuture</span><span class="o">(</span><span class="k">implicit</span> <span class="n">ec</span> <span class="k">=></span>
<span class="c1">// Create some Future using `ec`:</span>
<span class="o">???</span>
<span class="o">)</span></code></pre></figure>
<p>When this method is used to lift a <code class="language-plaintext highlighter-rouge">Future</code> into an effect, ZIO can manage where the <code class="language-plaintext highlighter-rouge">Future</code> is executed, and other methods like <code class="language-plaintext highlighter-rouge">evalOn</code> will correctly migrate the <code class="language-plaintext highlighter-rouge">Future</code> to the appropriate execution context.</p>
<p>Cats IO chose to accept a <code class="language-plaintext highlighter-rouge">Future</code> that has already been constructed with an external <code class="language-plaintext highlighter-rouge">ExecutionContext</code>. This means that Cats IO has no way of shifting the execution of an embedded <code class="language-plaintext highlighter-rouge">Future</code> to conform with the semantics of <code class="language-plaintext highlighter-rouge">evalOn</code> or <code class="language-plaintext highlighter-rouge">shift</code>. Moreover, it burdens the user of the API to choose an execution context for the <code class="language-plaintext highlighter-rouge">Future</code>, which means a fixed choice and separate plumbing.</p>
<p>Since one can always choose to ignore the provided <code class="language-plaintext highlighter-rouge">ExecutionContext</code>, the ZIO choice can be seen as a strict generalization of Cats IO capabilities, providing more seamless and precise interop with <code class="language-plaintext highlighter-rouge">Future</code> in the common case, but not preventing exceptions to the rule.</p>
<h1 id="8-blocking-io">8. Blocking IO</h1>
<p>As covered in <a href="/articles/zio-threads">Thread Pool Best Practices with ZIO</a>, server-side applications must have at least two separate thread pools for maximum efficiency:</p>
<ul>
<li>A fixed thread pool for CPU / async effects</li>
<li>A dynamic, growing thread pool for blocking effects</li>
</ul>
<p>A choice to run all effects on a fixed thread pool will eventually lead to deadlock; while a choice to run all effects on a dynamic, growing thread pool will lead to gross inefficiency.</p>
<p>On the JVM, ZIO provides two operators that provide direct support for blocking effects:</p>
<ul>
<li>The <code class="language-plaintext highlighter-rouge">blocking(effect)</code> operator, which will shift execution of the specified effect to a blocking thread pool, which uses very good settings and can also be configured;</li>
<li>The <code class="language-plaintext highlighter-rouge">effectBlocking(effect)</code> operator, which translates side-effectful blocking code into a pure effect, whose interruption will interrupt a lot of blocking code.</li>
</ul>
<p>If you have an effect, and you need to make sure it’s executed on a blocking thread pool, then you can wrap it in <code class="language-plaintext highlighter-rouge">blocking</code>. On the other hand, if you are wrapping some side-effectful code that blocks, then you can wrap it in <code class="language-plaintext highlighter-rouge">effectBlocking</code>, and benefit from ZIO’s composable, pervasive, and safe interruption (where possible).</p>
<p>Cats IO chose to adopt a more minimal core, and delegate such functionality to user-land code. While there are libraries that help provide the functionality of the <code class="language-plaintext highlighter-rouge">blocking</code> operator, they are based on <code class="language-plaintext highlighter-rouge">evalOn</code>, and therefore cannot actually guarantee execution on the blocking thread pool.</p>
<p>Power users may very well want to configure their own custom blocking thread pool (which of course, you can do with ZIO), or create more than these two thread pools (for example, a thread pool for low-latency event dispatching), but these operations provide exactly the desired semantics for the vast majority of cases.</p>
<h1 id="9-cost-free-effects">9. Cost-Free Effects</h1>
<p>Many functional Scala applications end up using one or both of the following monad transformers:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">ReaderT</code> / <code class="language-plaintext highlighter-rouge">Kleisli</code>, which adds the effect of accessing an environment</li>
<li><code class="language-plaintext highlighter-rouge">EitherT</code>, which adds the effect of typed errors (or <code class="language-plaintext highlighter-rouge">OptionT</code>, which is a specialization of <code class="language-plaintext highlighter-rouge">EitherT</code> with <code class="language-plaintext highlighter-rouge">Unit</code> as the failure type)</li>
</ul>
<p>The pattern is so pervasive, whole libraries have been designed around one or the other (for example, <em>http4s</em> extensively uses <code class="language-plaintext highlighter-rouge">Kleisli</code> and <code class="language-plaintext highlighter-rouge">OptionT</code>).</p>
<p>Using an advanced technique called <a href="/articles/rotating-effects">effect rotation</a>, ZIO provides both the <em>reader</em> and <em>typed error</em> capabilities directly in the <code class="language-plaintext highlighter-rouge">ZIO</code> data type.</p>
<p>Because not every user will need reader and typed error capabilities, ZIO also provides a variety of type / companion synonyms that cover common cases. For example, <code class="language-plaintext highlighter-rouge">Task[A]</code> provides only the core primary capability, without reader or typed errors.</p>
<p>This allows ZIO to provide the two most common (secondary) effects in functional applications without any runtime overhead whatsoever. In addition, supporting these effects directly in ZIO actually <em>reduced</em> the size of its runtime, allowing simpler code and pulling non-essential functionality out of the microkernel.</p>
<p>Cats IO chose to provide just a primary effect. This means that users who need reader or typed errors, or just want hack-free implementations of state, writer, and other type classes, will likely find themselves using monad transformers.</p>
<p>ZIO can be up to 8x faster than Cats IO with an equivalent effect stack. While the impact of effect overhead on application performance will depend on a great many factors, greater performance increases the number of applications for functional Scala, and allows developers to build their applications from fine-grained effects.</p>
<h1 id="10-microkernel-architecture">10. Microkernel Architecture</h1>
<p>ZIO utilizes a microkernel architecture, which pulls as much functionality as possible out of the runtime system, and into ordinary user-land code, written in pure functional Scala. Indeed, even parts of the microkernel are itself written in pure functional Scala, utilizing an even smaller core for bootstrapping.</p>
<p>While the original ZIO kernel was roughly 2,000 lines of code, after introducing typed errors and environment, and eliminating redundancy and improving orthogonality, the entire microkernel is now 375 SLOC, in a single file.</p>
<p>As the complexity of modern effect systems in Scala has grown, so has the potential for bugs. There are very few people in the world who understand how these systems work, and the potential for hidden bugs and edge cases is very high.</p>
<p>Personally, I am a fan of microkernel effect systems for the following reasons:</p>
<ul>
<li>The smaller code can be more easily inspected for correctness</li>
<li>There are fewer places for bugs and edge-cases to hide</li>
<li>It is cheaper, faster, and safer to respond to real world feedback</li>
<li>It is easier for new contributors to help out with core maintenance</li>
</ul>
<p>Monolithic kernels can in theory be much better optimized. However, due to the volunteer nature of open source, we only have finite resources for optimization.</p>
<p>Due to these constraints, often you can either micro-optimize a <em>part</em> of a monolithic kernel, or micro-optimize the <em>whole</em> of a microkernel. The former can give you super high-performance in a few cases, while the latter can give you great performance across a wide range of complex cases.</p>
<p>Of all the effect systems out there, the ZIO runtime is by far the smallest. As a reference implementation, Cats IO comes in second place, but its runtime is at least <em>twice</em> the size of the ZIO runtime (maybe <em>three</em> times, depending on how you count).</p>
<h1 id="11-beginner-friendly">11. Beginner-Friendly</h1>
<p>ZIO has made many decisions to increase usability for new users, without cutting corners or sacrificing principles for advanced users. For example:</p>
<ul>
<li>Jargon-free naming. For example:
<ul>
<li><code class="language-plaintext highlighter-rouge">ZIO.succeed</code> instead of <code class="language-plaintext highlighter-rouge">Applicative[F].pure</code></li>
<li><code class="language-plaintext highlighter-rouge">zip</code> instead of <code class="language-plaintext highlighter-rouge">Apply[F].product</code></li>
<li><code class="language-plaintext highlighter-rouge">ZIO.foreach</code> instead of <code class="language-plaintext highlighter-rouge">Traverse[F].traverse</code></li>
<li>Etc.</li>
</ul>
</li>
<li>No use of higher-kinded types or type classes (Cats, Cats Effect, and Scalaz instances are available in optional modules)</li>
<li>No implicits that have to be imported or summoned (except for <code class="language-plaintext highlighter-rouge">Runtime</code>, which must be implicit for all Cats Effect projects, due to the current design of Cats Effect); implicits are a constant source of frustration for new Cats IO users</li>
<li>No required syntax classes</li>
<li>Auto-complete-friendly naming that groups similar methods by prefix. For example:
<ul>
<li><code class="language-plaintext highlighter-rouge">zip</code> / <code class="language-plaintext highlighter-rouge">zipPar</code></li>
<li><code class="language-plaintext highlighter-rouge">ZIO.foreach</code> / <code class="language-plaintext highlighter-rouge">ZIO.foreachPar</code></li>
<li><code class="language-plaintext highlighter-rouge">ZIO.succeed</code> / <code class="language-plaintext highlighter-rouge">ZIO.succeedLazy</code></li>
<li>etc.</li>
</ul>
</li>
<li>Concrete methods on concrete data types, which aids discoverability and traversability, and makes ZIO very usable in IDEs</li>
<li>Conversion from all Scala data types to the ZIO effect type
<ul>
<li><code class="language-plaintext highlighter-rouge">ZIO.fromFuture</code></li>
<li><code class="language-plaintext highlighter-rouge">ZIO.fromOption</code></li>
<li><code class="language-plaintext highlighter-rouge">ZIO.fromEither</code></li>
<li><code class="language-plaintext highlighter-rouge">ZIO.fromTry</code></li>
<li>etc.</li>
</ul>
</li>
<li>Full, out-of-the-box type inference for all data types and methods</li>
</ul>
<p>Anecdotally, I have seen people with no prior background in functional Scala successfully build prototypes using ZIO without any external assistance, and before there was any good documentation—unaware that by using ZIO, they were writing purely functional code.</p>
<p>Cats IO chose to delegate most functionality, names, and decisions around type-inference to Cats. This keeps the reference implementation small, but may increase ramp-up time for developers new to functional programming, and result in well-known usability problems around discoverability, naming, implicits, and type inference.</p>
<h1 id="12-batteries-included">12. Batteries Included</h1>
<p>In a small, cross-platform package, ZIO provides a highly-integrated toolbox for building principled asynchronous and concurrent applications.</p>
<p>This toolbox includes the following:</p>
<ul>
<li>The most important concurrent data structures, including <code class="language-plaintext highlighter-rouge">Ref</code>, <code class="language-plaintext highlighter-rouge">Promise</code>, <code class="language-plaintext highlighter-rouge">Queue</code>, <code class="language-plaintext highlighter-rouge">Semaphore</code>, and a small <code class="language-plaintext highlighter-rouge">Stream</code> for file / socket / data streaming</li>
<li><a href="https://www.youtube.com/watch?list=PL8NC5lCgGs6MYG0hR_ZOhQLvtoyThURka&v=d6WWmia0BPM">Software Transactional Memory</a> (STM), which can be used to simply build composable, asynchronous, concurrent, and interruptible data structures</li>
<li><code class="language-plaintext highlighter-rouge">Schedule</code>, which offers composable retries and repetitions</li>
<li>Tiny and testable <code class="language-plaintext highlighter-rouge">Clock</code>, <code class="language-plaintext highlighter-rouge">Random</code>, <code class="language-plaintext highlighter-rouge">Console</code>, and <code class="language-plaintext highlighter-rouge">System</code> services, which are used by nearly every application</li>
<li>Many helper methods on the effect type covering common use cases</li>
</ul>
<p>As a reference implementation, Cats IO has none of these features. This decision makes Cats IO more lightweight, but at the cost of adding more third-party dependencies (where they are available), or having to write more user-land code.</p>
<h1 id="summary">Summary</h1>
<p>Cats Effect has done great things for the Scala ecosystem, providing a growing roster of libraries that all work together.</p>
<p>Application developers who are using Cats Effect libraries now face the difficult decision of choosing which of the major effect types to use with Cats Effect libraries: Cats IO, Monix, or ZIO.</p>
<p>While different people will make different choices that are uniquely suited for them, if you value some of the design decisions described in this post, then I hope you will find that together, ZIO and Cats Effect make a killer combination!</p>
<p><a href="https://degoes.net/articles/zio-cats-effect">ZIO & Cats Effect: A Match Made in Heaven</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on April 18, 2019.</p>https://degoes.net/articles/zio-challenge2019-03-10T00:00:00-07:002019-03-10T00:00:00-07:00John A De Goeshttps://degoes.netjohn@degoes.net<p>Recently, while teaching my first-ever workshop for <a href="https://github.com/scalaz/scalaz-zio">ZIO</a> with the great crew at <a href="https://scalac.io">Scalac</a>, we had a chance to build a purely functional circuit breaker using ZIO’s <code class="language-plaintext highlighter-rouge">Ref</code>, which is a model of a mutable reference that can be updated atomically.</p>
<p>A circuit breaker guards access to an external service, like a database, third-party API or microservice. Once too many requests to the service fail, the circuit breaker trips, and immediately fails all requests, until the circuit breaker has a chance to reset.</p>
<p>Circuit breakers not only protect external services from overload (giving them a chance to recover after failure), but they help conserve local resources (such as sockets, threads, and the like) that would otherwise be wasted on a lost cost.</p>
<p>As opposed to retry policies, which dictate how individual requests are retried, circuit breakers share global knowledge across a system, so different fibers can act more intelligently and in a coordinated fashion.</p>
<p>Circuit breakers are often modeled as having three states: open, closed, and half-open. The circuit breaker logic (possibly aided by configuration parameters) is responsible for transitioning between the states based on inspecting the status of requests.</p>
<p>At the ZIO workshop, exploring different possibilities for circuit breakers made me realize something: I <em>really</em> don’t like circuit breakers. I find the arbitrary nature of the number of states and the switching conditions deeply disturbing.</p>
<p>I think we can do better than circuit breakers, and have some fun while we’re at it! So in this post, I’m going to issue a challenge for all you fans of functional programming in Scala: build a better circuit breaker!</p>
<h2 id="the-challenge">The Challenge</h2>
<p>Instead of a circuit breaker, I want you to build a <em>tap</em>, which adjusts the flow of requests continuously through the tap.</p>
<p>The flow is adjusted based on observed failures that qualify (i.e. match some user-defined predicate).</p>
<p>If you want to use ZIO to implement the <code class="language-plaintext highlighter-rouge">Tap</code>, then your API should conform to the following interface:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="cm">/**
* A `Tap` adjusts the flow of tasks through
* an external service in response to observed
* failures in the service, always trying to
* maximize flow while attempting to meet the
* user-defined upper bound on failures.
*/</span>
<span class="k">trait</span> <span class="nc">Tap</span><span class="o">[</span><span class="kt">-E1</span>, <span class="kt">+E2</span><span class="o">]</span> <span class="o">{</span>
<span class="cm">/**
* Sends the task through the tap. The
* returned task may fail immediately with a
* default error depending on the service
* being guarded by the tap.
*/</span>
<span class="k">def</span> <span class="nf">apply</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span> <span class="k">>:</span> <span class="kt">E2</span> <span class="k"><:</span> <span class="kt">E1</span>, <span class="kt">A</span><span class="o">](</span>
<span class="n">effect</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">A</span><span class="o">])</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">A</span><span class="o">]</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">Tap</span> <span class="o">{</span>
<span class="cm">/**
* Creates a tap that aims for the specified
* maximum error rate, using the specified
* function to qualify errors (unqualified
* errors are not treated as failures for
* purposes of the tap), and the specified
* default error used for rejecting tasks
* submitted to the tap.
*/</span>
<span class="k">def</span> <span class="nf">make</span><span class="o">[</span><span class="kt">E1</span>, <span class="kt">E2</span><span class="o">](</span>
<span class="n">errBound</span> <span class="k">:</span> <span class="kt">Percentage</span><span class="o">,</span>
<span class="n">qualified</span> <span class="k">:</span> <span class="kt">E1</span> <span class="o">=></span> <span class="nc">Boolean</span><span class="o">,</span>
<span class="n">rejected</span> <span class="k">:</span> <span class="o">=></span> <span class="n">E2</span><span class="o">)</span><span class="k">:</span> <span class="kt">UIO</span><span class="o">[</span><span class="kt">Tap</span><span class="o">[</span><span class="kt">E1</span>, <span class="kt">E2</span><span class="o">]]</span> <span class="k">=</span> <span class="o">???</span>
<span class="o">}</span></code></pre></figure>
<p>If you want to use Cats IO or Monix to implement <code class="language-plaintext highlighter-rouge">Tap</code>, then your API should conform to the following interface (or its polymorphic equivalent):</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">trait</span> <span class="nc">Tap</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">apply</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">effect</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">A</span><span class="o">])</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">Tap</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">make</span><span class="o">(</span>
<span class="n">errBound</span> <span class="k">:</span> <span class="kt">Percentage</span><span class="o">,</span>
<span class="n">qualified</span> <span class="k">:</span> <span class="kt">Throwable</span> <span class="o">=></span> <span class="nc">Boolean</span><span class="o">,</span>
<span class="n">rejected</span> <span class="k">:</span> <span class="o">=></span> <span class="nc">Throwable</span><span class="o">)</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">Tap</span><span class="o">]</span> <span class="k">=</span> <span class="o">???</span>
<span class="o">}</span></code></pre></figure>
<p>Your implementation of <code class="language-plaintext highlighter-rouge">Tap</code> should satisfy the following requirement:</p>
<p><em>The tap must continuously adjust the percentage of tasks it lets through until it finds the maximum flow rate that satisfies the user-defined maximum error bound.</em></p>
<p>Thus, if you create a tap with a maximum error rate of <em>1%</em>, and suddenly 50% of all tasks are failing, then the tap will decrease flow until the failure rate stabilizes at 1%.</p>
<p>As the service is recovering, the failure rate will drop below 1%, which will cause the tap to increase flow and let more tasks through.</p>
<p>Once the service has fully recovered, the failure rate will hit 0% (or within some distance of that target), at which point, the tap will let all tasks through.</p>
<p>Your implementation must be purely functional and concurrent. Bonus points for demonstrating your knowledge of concurrency primitives, such as <code class="language-plaintext highlighter-rouge">Ref</code>, <code class="language-plaintext highlighter-rouge">Promise</code> (<code class="language-plaintext highlighter-rouge">Deferred</code>), and so forth.</p>
<h2 id="winners">Winners</h2>
<p>The main reason to work on this challenge is to explore solutions for concurrency in functional Scala. It’s a fun little project that will take you on a grand tour of modern, purely functional effect systems in Scala.</p>
<p>That said, I want to give you a little extra motivation to work on this problem!</p>
<p>If you post your code in a Gist so the whole world can learn from your solution, then I’ll both promote your solution, and buy you a drink next time we’re in the same city!</p>
<p>Finally, if your solution is among the top 1-3 I receive over the next 2 weeks, I’ll connect with you on LinkedIn and write a short, honest endorsement of your skills in functional Scala.</p>
<p>Ready? On your marks, get set, go!</p>
<p><a href="https://degoes.net/articles/zio-challenge">The Functional Scala Concurrency Challenge</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on March 10, 2019.</p>https://degoes.net/articles/testable-zio2019-03-07T00:00:00-07:002019-03-07T00:00:00-07:00John A De Goeshttps://degoes.netjohn@degoes.net<p>In my <a href="/articles/zio-environment">last post</a>, I introduced <em>ZIO Environment</em>, which is a new feature in ZIO that bakes in a high-performance, type-safe, and fully-inferred reader effect into the ZIO data type.</p>
<p>This capability leads to a way of describing and testing effects that I call <em>environmental effects</em>. Unlike tagless-final, which is difficult to teach, difficult to abstract over, and does not infer, environmental effects are simple, abstract well, and infer completely.</p>
<p>Moreover, while proponents of tagless-final argue that tagless-final <em>parametrically constrains</em> effects, my <a href="/articles/zio-environment">last post</a> demonstrated this is not quite correct: not only can you embed raw effects <em>anywhere</em> in Scala, but even without leaving <em>Scalazzi</em> (the purely functional subset of Scala), you can lift arbitrary effects into any <code class="language-plaintext highlighter-rouge">Applicative</code> functor.</p>
<p>The inability of tagless-final to constrain effects is more than just theoretical:</p>
<ul>
<li>New Scala functional programmers use effect type classes like <code class="language-plaintext highlighter-rouge">Sync</code> everywhere (which are themselves lawless and serve only to embed effects), and they embed effects using lazy methods, like <code class="language-plaintext highlighter-rouge">defer</code> or <code class="language-plaintext highlighter-rouge">point</code>.</li>
<li>Even some experienced Scala functional programmers embed effects in pure methods (for example, exceptions in the functions they pass to <code class="language-plaintext highlighter-rouge">map</code>, <code class="language-plaintext highlighter-rouge">flatMap</code>), and some effect types encourage this behavior.</li>
</ul>
<p>Tagless-final can be used by a well-trained and highly-disciplined team to constrain effects, but the same can be said for many approaches, <em>including</em> environmental effects.</p>
<p>After professionally and pedagogically wrestling with these issues for several years now, I’ve come to the conclusion there are just two legitimately compelling reasons to use tagless-final:</p>
<ol>
<li>Avoiding commitment to a specific effect type, which can be useful for library authors, but which is less useful for application developers (often it’s a hinderance!);</li>
<li>Writing testable functional code, which is fairly straightforward with tagless-final because you can just create test instances for different effect type classes.</li>
</ol>
<p>While testability is a compelling reason to <em>use</em> tagless-final, it’s not necessarily a compelling reason to <em>choose</em> tagless-final over other approaches—in particular, over environmental effects.</p>
<p>In this post, I’m going to show you how to use environmental effects to achieve testability. I hope to demonstrate that environmental effects provide easier and more incremental testability—all without sacrificing teachability, abstraction, or type inference.</p>
<h2 id="a-web-app">A Web App</h2>
<p>Let’s say we are building a web application with ZIO. Suppose the application was originally written with <code class="language-plaintext highlighter-rouge">Future</code> or perhaps some version of <code class="language-plaintext highlighter-rouge">IO</code> or <code class="language-plaintext highlighter-rouge">Task</code>.</p>
<p>Later, the application was ported to ZIO’s <code class="language-plaintext highlighter-rouge">Task[A]</code>, which is a type alias for <code class="language-plaintext highlighter-rouge">ZIO[Any, Throwable, A]</code>—representing an effect that requires no specific environment and that may fail with any <code class="language-plaintext highlighter-rouge">Throwable</code>.</p>
<p>Now let’s say one of the functions in our application, called <code class="language-plaintext highlighter-rouge">inviteFriends</code>, invites the friends of a given user to the application by sending them emails:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">inviteFriends</span><span class="o">(</span><span class="n">userID</span><span class="k">:</span> <span class="kt">UserID</span><span class="o">)</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">Send</span><span class="o">]]</span> <span class="k">=</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">user</span> <span class="k"><-</span> <span class="nv">DB</span><span class="o">.</span><span class="py">lookupUser</span><span class="o">(</span><span class="n">userID</span><span class="o">)</span>
<span class="n">friends</span> <span class="k"><-</span> <span class="nv">Social</span><span class="o">.</span><span class="py">getFriends</span><span class="o">(</span><span class="nv">user</span><span class="o">.</span><span class="py">facebookID</span><span class="o">)</span>
<span class="n">resp</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">foreach</span><span class="o">(</span><span class="n">friends</span><span class="o">)</span> <span class="o">{</span> <span class="n">friend</span> <span class="k">=></span>
<span class="nv">Email</span><span class="o">.</span><span class="py">invite</span><span class="o">(</span><span class="n">user</span><span class="o">,</span> <span class="nv">friend</span><span class="o">.</span><span class="py">email</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span> <span class="k">yield</span> <span class="n">resp</span></code></pre></figure>
<p>Portions of the <code class="language-plaintext highlighter-rouge">Social</code>, <code class="language-plaintext highlighter-rouge">DB</code>, and <code class="language-plaintext highlighter-rouge">Email</code> objects are shown below:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">object</span> <span class="nc">Social</span> <span class="o">{</span>
<span class="o">...</span>
<span class="k">def</span> <span class="nf">getFriends</span><span class="o">(</span><span class="n">fid</span><span class="k">:</span> <span class="kt">FacebookID</span><span class="o">)</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">FacebookProfile</span><span class="o">]]</span> <span class="k">=</span> <span class="o">???</span>
<span class="o">...</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">DB</span> <span class="o">{</span>
<span class="o">...</span>
<span class="k">def</span> <span class="nf">lookupUser</span><span class="o">(</span><span class="n">uid</span><span class="k">:</span> <span class="kt">UserID</span><span class="o">)</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">UserProfile</span><span class="o">]</span> <span class="k">=</span> <span class="o">???</span>
<span class="o">...</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">Email</span> <span class="o">{</span>
<span class="o">...</span>
<span class="k">def</span> <span class="nf">invite</span><span class="o">(</span><span class="n">user</span><span class="k">:</span> <span class="kt">UserProfile</span><span class="o">,</span> <span class="n">friend</span><span class="k">:</span> <span class="kt">FacebookProfile</span><span class="o">)</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">Send</span><span class="o">]</span> <span class="k">=</span> <span class="o">???</span>
<span class="o">...</span>
<span class="o">}</span></code></pre></figure>
<p>As currently written, our web application is not very testable. The function <code class="language-plaintext highlighter-rouge">inviteFriends</code> makes direct calls to database functions, Facebook API functions, and email service functions.</p>
<p>While we may have automated tests for our web service, because our application interacts directly with the real world, the tests are actually <em>system tests</em>, not <em>unit tests</em>. Such tests are very difficult to write, they run slowly, they randomly fail, and they test much more than our application logic.</p>
<p>We do not have time to rewrite our application, and we cannot make it testable all at once. Instead, let’s try to remove dependency on the live database for the <code class="language-plaintext highlighter-rouge">inviteFriends</code> function.</p>
<p>If we succeed in doing this, we will make our test code a <em>little better</em>, and after we ship the new code, we can incrementally use the same technique to make the function <em>fully</em> testable—fast, deterministic, and without any external dependencies.</p>
<h3 id="steps-toward-testability">Steps Toward Testability</h3>
<p>To incrementally refactor <code class="language-plaintext highlighter-rouge">inviteFriends</code> to be more testable, we’re going to perform the following series of refactorings:</p>
<ol>
<li>Introduce a type alias.</li>
<li>Introduce a module for the database.</li>
<li>Implement a production database module.</li>
<li>Integrate the production module.</li>
<li>Implement a test database module.</li>
<li>Test the <code class="language-plaintext highlighter-rouge">inviteFriends</code> function.</li>
</ol>
<p>Each of these steps will be covered in the sections that follow.</p>
<h3 id="introduce-a-type-alias">Introduce A Type Alias</h3>
<p>To simplify the process of refactoring our application, we’re going to first introduce a simple type alias that we can use in the definition of <code class="language-plaintext highlighter-rouge">inviteFriends</code> and the functions that call it:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">type</span> <span class="kt">Webapp</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Task</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span></code></pre></figure>
<p>Now we will mechanically update the <code class="language-plaintext highlighter-rouge">lookupUser</code> function and any functions that call it to use the type alias:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">inviteFriends</span><span class="o">(</span><span class="n">userID</span><span class="k">:</span> <span class="kt">UserID</span><span class="o">)</span><span class="k">:</span> <span class="kt">Webapp</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">Send</span><span class="o">]]</span> <span class="k">=</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">user</span> <span class="k"><-</span> <span class="nv">DB</span><span class="o">.</span><span class="py">lookupUser</span><span class="o">(</span><span class="n">userID</span><span class="o">)</span>
<span class="n">friends</span> <span class="k"><-</span> <span class="nv">Social</span><span class="o">.</span><span class="py">getFriends</span><span class="o">(</span><span class="nv">user</span><span class="o">.</span><span class="py">facebookID</span><span class="o">)</span>
<span class="n">resp</span> <span class="k"><-</span> <span class="nv">ZIO</span><span class="o">.</span><span class="py">foreach</span><span class="o">(</span><span class="n">friends</span><span class="o">)</span> <span class="o">{</span> <span class="n">friend</span> <span class="k">=></span>
<span class="nv">Email</span><span class="o">.</span><span class="py">invite</span><span class="o">(</span><span class="n">user</span><span class="o">,</span> <span class="nv">friend</span><span class="o">.</span><span class="py">email</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span> <span class="k">yield</span> <span class="n">resp</span></code></pre></figure>
<p>As an alternative to this technique, we could simply delete the return types entirely. However, it’s a good practice to place return types on top-level function signatures, so developers without IDEs can easily determine the return type of functions.</p>
<p>After this step, we are ready to introduce a service for the database.</p>
<h3 id="introduce-a-database-module">Introduce a Database Module</h3>
<p>The database module will provide access to a database service.</p>
<p>As discussed in my post on <a href="/articles/zio-environment">ZIO Environment</a>, the database module is an ordinary interface with a single field, which contains the database service.</p>
<p>We can define both the module and the service very simply:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="c1">// The database module</span>
<span class="k">trait</span> <span class="nc">Database</span> <span class="o">{</span>
<span class="k">val</span> <span class="nv">database</span><span class="k">:</span> <span class="kt">Database.Service</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">Database</span> <span class="o">{</span>
<span class="c1">// The database service</span>
<span class="k">trait</span> <span class="nc">Service</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">lookupUser</span><span class="o">(</span><span class="n">uid</span><span class="k">:</span> <span class="kt">UserID</span><span class="o">)</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">UserProfile</span><span class="o">]</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></figure>
<p>Notice how we have decided to place just one method inside the database service: the <code class="language-plaintext highlighter-rouge">lookupUser</code> method. Although there may be many database methods, we don’t have time to make all of them testable, so we will focus on the one required by the <code class="language-plaintext highlighter-rouge">inviteFriends</code> method.</p>
<p>We are now ready to implement a production version of the service.</p>
<h3 id="implement-production-module">Implement Production Module</h3>
<p>We will call the production database module <code class="language-plaintext highlighter-rouge">DatabaseLive</code>. To implement the module, we need only copy and paste the implementation of <code class="language-plaintext highlighter-rouge">Database.lookupUser</code> into our implementation of the service interface:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">trait</span> <span class="nc">DatabaseLive</span> <span class="k">extends</span> <span class="nc">Database</span> <span class="o">{</span>
<span class="k">val</span> <span class="nv">database</span> <span class="k">=</span>
<span class="k">new</span> <span class="nv">Database</span><span class="o">.</span><span class="py">Service</span> <span class="o">{</span>
<span class="c1">// Implementation copy/pasted from</span>
<span class="c1">// DB.lookupUser:</span>
<span class="k">def</span> <span class="nf">lookupUser</span><span class="o">(</span><span class="n">userID</span><span class="k">:</span> <span class="kt">UserID</span><span class="o">)</span> <span class="k">=</span>
<span class="o">...</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">DatabaseLive</span> <span class="k">extends</span> <span class="nc">DatabaseLive</span></code></pre></figure>
<p>For maximum flexibility and convenience, we have defined both a <em>trait</em> that implements the database module, which can be mixed into other traits, and an <em>object</em> that extends the trait, which can be used standalone.</p>
<h3 id="integrate-production-module">Integrate Production Module</h3>
<p>We now have all the pieces we need to replace the original <code class="language-plaintext highlighter-rouge">DB.lookupUser</code> method, whose actual implementation now resides inside our <code class="language-plaintext highlighter-rouge">DatabaseLive</code> module:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">object</span> <span class="nc">DB</span> <span class="o">{</span>
<span class="o">...</span>
<span class="k">def</span> <span class="nf">lookupUser</span><span class="o">(</span><span class="n">uid</span><span class="k">:</span> <span class="kt">UserID</span><span class="o">)</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">Database</span>, <span class="kt">Throwable</span>, <span class="kt">UserProfile</span><span class="o">]</span> <span class="k">=</span>
<span class="nv">ZIO</span><span class="o">.</span><span class="py">accessM</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">database</span> <span class="n">lookupUser</span> <span class="n">uid</span><span class="o">)</span>
<span class="o">...</span>
<span class="o">}</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">lookupUser</code> method merely delegates to the database module, by accessing the model through ZIO Environment (<code class="language-plaintext highlighter-rouge">ZIO.accessM</code>).</p>
<p>Here we don’t use the <code class="language-plaintext highlighter-rouge">Webapp</code> type alias, because the functions in <code class="language-plaintext highlighter-rouge">DB</code> will not necessarily have the same dependencies as our web application.</p>
<p>However, after enough refactoring, we might introduce a new type alias in the <code class="language-plaintext highlighter-rouge">DB</code> object: <code class="language-plaintext highlighter-rouge">type DB[A] = ZIO[Database, Throwable, A]</code>. Eventually, all methods in <code class="language-plaintext highlighter-rouge">DB</code> might return effects of this type.</p>
<p>At this point, our refactoring is nearly complete. But we have to take care of one last detail: we have to provide our database module to the production application.</p>
<p>There are two main ways to provide the database module to our application. If it is inconvenient to propagate the <code class="language-plaintext highlighter-rouge">Webapp</code> type signature to the top of our application, we can always supply the production module somewhere inside our application.</p>
<p>In the worst case, if we are pressed for time and need to ship code today, maybe we choose to provide the production database wherever we call <code class="language-plaintext highlighter-rouge">inviteFriends</code>.</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="nf">inviteFriends</span><span class="o">(</span><span class="n">userId</span><span class="o">).</span><span class="py">provide</span><span class="o">(</span><span class="nc">DatabaseLive</span><span class="o">)</span></code></pre></figure>
<p>If we have a bit more time, we can push the <code class="language-plaintext highlighter-rouge">Webapp</code> type synonym to the entry point of our purely functional application, which might be the main function, or it might be where our web framework calls into our code.</p>
<p>In this case, instead of using the <code class="language-plaintext highlighter-rouge">DefaultRuntime</code> that ships with ZIO, we can define our own <code class="language-plaintext highlighter-rouge">Runtime</code>, which provides the production database module):</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">myRuntime</span> <span class="k">=</span>
<span class="nc">Runtime</span><span class="o">(</span><span class="nc">DatabaseLive</span><span class="o">,</span> <span class="nc">PlatformLive</span><span class="o">)</span></code></pre></figure>
<p>The custom runtime can be used to run many different effects that all require the same environment, so we don’t have to call <code class="language-plaintext highlighter-rouge">provide</code> on all of them before we run them.</p>
<p>Once we have this custom runtime, we can run our top-level effect, which will supply its required environment:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="nv">myRuntime</span><span class="o">.</span><span class="py">unsafeRun</span><span class="o">(</span><span class="n">effect</span><span class="o">)</span></code></pre></figure>
<p>At this point, we have not changed the behavior of our application at all—it will work exactly as it did before. We’ve just moved the code around a bit, so we can access a tiny effect through ZIO environment.</p>
<p>Now it’s time to build a database module specifically for testing.</p>
<h3 id="implement-test-module">Implement Test Module</h3>
<p>We could implement the test database module using a mocking framework. However, to avoid all magic and use of reflection, in this post, we will build one from scratch.</p>
<p>For maximum flexibility, our test database module will track all calls to <code class="language-plaintext highlighter-rouge">lookupUser</code>, and supply responses using a <code class="language-plaintext highlighter-rouge">Map</code>, which can be dynamically changed by the test suite.</p>
<p>To support this stateful behavior, we will need a <code class="language-plaintext highlighter-rouge">Ref</code>, which is a concurrent-safe ZIO data structure that models mutable references. We will also need a simple (immutable) data structure to hold the state of the test database module.</p>
<p>We define the following test data structure, which is capable of tracking a list of <code class="language-plaintext highlighter-rouge">UserID</code> values, and holding data that maps from <code class="language-plaintext highlighter-rouge">UserID</code> to <code class="language-plaintext highlighter-rouge">UserProfile</code>.</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">TestDatabaseState</span><span class="o">(</span>
<span class="n">lookups</span> <span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">UserID</span><span class="o">],</span>
<span class="n">data</span> <span class="k">:</span> <span class="kt">Map</span><span class="o">[</span><span class="kt">UserID</span>, <span class="kt">UserProfile</span><span class="o">]</span>
<span class="o">)</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">addLookup</span><span class="o">(</span><span class="n">uid</span><span class="k">:</span> <span class="kt">UserID</span><span class="o">)</span><span class="k">:</span> <span class="kt">TestDatabaseState</span> <span class="o">=</span> <span class="nf">copy</span><span class="o">(</span><span class="n">lookups</span> <span class="k">=</span> <span class="n">uid</span> <span class="o">::</span> <span class="n">lookups</span><span class="o">)</span>
<span class="o">}</span></code></pre></figure>
<p>Now we can define the service of our test database module. The service will require a <code class="language-plaintext highlighter-rouge">Ref[TestDatabaseState]</code>, so it can not only use test data, but update the test state:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">TestDatabaseService</span><span class="o">(</span><span class="n">ref</span><span class="k">:</span> <span class="kt">Ref</span><span class="o">[</span><span class="kt">TestDatabaseState</span><span class="o">])</span> <span class="k">extends</span> <span class="nv">Database</span><span class="o">.</span><span class="py">Service</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">lookupUser</span><span class="o">(</span><span class="n">uid</span><span class="k">:</span> <span class="kt">UserID</span><span class="o">)</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">UserProfile</span><span class="o">]</span> <span class="k">=</span>
<span class="k">for</span> <span class="o">{</span>
<span class="k">_</span> <span class="k"><-</span> <span class="nv">ref</span><span class="o">.</span><span class="py">update</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">addLookup</span> <span class="n">uid</span><span class="o">)</span>
<span class="n">data</span> <span class="k"><-</span> <span class="nv">ref</span><span class="o">.</span><span class="py">get</span><span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">data</span><span class="o">)</span>
<span class="n">profile</span> <span class="k"><-</span> <span class="nv">Task</span><span class="o">.</span><span class="py">fromEither</span><span class="o">(</span><span class="nv">data</span><span class="o">.</span><span class="py">get</span><span class="o">(</span><span class="n">uid</span><span class="o">)</span>
<span class="o">.</span><span class="py">fold</span><span class="o">(</span><span class="nc">Left</span><span class="o">(</span><span class="k">new</span> <span class="nc">DBErr</span><span class="o">))(</span>
<span class="nc">Right</span><span class="o">(</span><span class="k">_</span><span class="o">)))</span>
<span class="o">}</span> <span class="k">yield</span> <span class="n">profile</span>
<span class="o">}</span></code></pre></figure>
<p>Notice how the <code class="language-plaintext highlighter-rouge">lookupUser</code> function stores the <code class="language-plaintext highlighter-rouge">UserID</code> of every call in the <code class="language-plaintext highlighter-rouge">lookups</code> field of the <code class="language-plaintext highlighter-rouge">TestDatabaseState</code>. In addition, the function retrieves test responses from the map. If there is no response in the map, the function fails, presumably in the same way the production database would fail.</p>
<p>The test service must be placed in a module. In general, we should wait to create the module until the test suite, because then we will know the full set of dependencies for each test.</p>
<p>However, at this stage, the database service is the only dependency in our application, so we can make a helper function to create the test database module:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">object</span> <span class="nc">TestDatabase</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">apply</span><span class="o">(</span><span class="n">ref</span><span class="k">:</span> <span class="kt">Ref</span><span class="o">[</span><span class="kt">TestDatabaseState</span><span class="o">])</span><span class="k">:</span> <span class="kt">Database</span> <span class="o">=</span>
<span class="k">new</span> <span class="nc">Database</span> <span class="o">{</span>
<span class="k">val</span> <span class="nv">database</span><span class="k">:</span> <span class="kt">Database.Service</span> <span class="o">=</span>
<span class="k">new</span> <span class="nc">TestDatabaseService</span><span class="o">(</span><span class="n">ref</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></figure>
<p>We now have all the pieces necessary to write a test of the <code class="language-plaintext highlighter-rouge">inviteFriends</code> function, which will use our test database module.</p>
<h3 id="write-the-test">Write the Test</h3>
<p>To more easily test the <code class="language-plaintext highlighter-rouge">lookupFriends</code> function, we will define a helper function. Given test data and input to the function, the helper will return the final test state and the output of the <code class="language-plaintext highlighter-rouge">lookupFriends</code> function:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">def</span> <span class="nf">runLookupFriends</span><span class="o">(</span><span class="n">data</span><span class="k">:</span> <span class="kt">Map</span><span class="o">[</span><span class="kt">UserID</span>, <span class="kt">UserProfile</span><span class="o">],</span> <span class="n">uid</span><span class="k">:</span> <span class="kt">UserID</span><span class="o">)</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[(</span><span class="kt">TestDatabaseState</span>, <span class="kt">List</span><span class="o">[</span><span class="kt">Send</span><span class="o">])]</span> <span class="k">=</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">ref</span> <span class="k"><-</span> <span class="nv">Ref</span><span class="o">.</span><span class="py">make</span><span class="o">(</span><span class="nc">TestDatabaseState</span><span class="o">(</span><span class="nc">Nil</span><span class="o">,</span> <span class="n">data</span><span class="o">))</span>
<span class="n">resp</span> <span class="k"><-</span> <span class="nf">lookupFriends</span><span class="o">(</span><span class="n">uid</span><span class="o">)</span>
<span class="o">.</span><span class="py">provide</span><span class="o">(</span><span class="nc">TestDatabase</span><span class="o">(</span><span class="n">ref</span><span class="o">))</span>
<span class="n">state</span> <span class="k"><-</span> <span class="nv">ref</span><span class="o">.</span><span class="py">get</span>
<span class="o">}</span> <span class="nf">yield</span> <span class="o">(</span><span class="n">state</span><span class="o">,</span> <span class="n">resp</span><span class="o">)</span></code></pre></figure>
<p>The helper function creates a <code class="language-plaintext highlighter-rouge">Ref</code> with the initial test data, uses the <code class="language-plaintext highlighter-rouge">Ref</code> to create the <code class="language-plaintext highlighter-rouge">TestDatabase</code> module, and then supplies the database module to the effect returned by <code class="language-plaintext highlighter-rouge">lookupFriends</code>.</p>
<p>With this helper function, writing a test becomes quite simple:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">class</span> <span class="nc">TestSuite</span> <span class="k">extends</span> <span class="nc">DefaultRuntime</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">testLookupFriends</span> <span class="k">=</span> <span class="o">{</span>
<span class="nf">val</span> <span class="o">(</span><span class="n">state</span><span class="o">,</span> <span class="n">resp</span><span class="o">)</span> <span class="k">=</span>
<span class="n">unsafeRun</span> <span class="o">{</span>
<span class="nf">runLookupFriends</span><span class="o">(</span>
<span class="nc">Map</span><span class="o">(</span><span class="nc">TestUserID</span> <span class="o">-></span> <span class="nc">TestUserProfile</span><span class="o">),</span>
<span class="nc">TestUserID</span>
<span class="o">)</span>
<span class="o">}</span>
<span class="o">(</span><span class="nv">state</span><span class="o">.</span><span class="py">lookups</span> <span class="n">must_===</span> <span class="nc">List</span><span class="o">(</span><span class="nc">TestUserID</span><span class="o">))</span> <span class="nf">and</span>
<span class="o">(</span><span class="n">resp</span> <span class="n">must_===</span> <span class="nc">TestResponse</span><span class="o">)</span>
<span class="o">}</span></code></pre></figure>
<p>This test for <code class="language-plaintext highlighter-rouge">inviteFriends</code> is not perfect. It still interacts with a real Facebook API and a real email service. But compared to whatever tests already exist, at least this test does not interact with a real database.</p>
<p>Moreover, we were able to make this change in a <em>minimally disruptive</em> manner.</p>
<h3 id="a-glimpse-beyond">A Glimpse Beyond</h3>
<p>After a little more refactoring, of course, we would succeed in making <code class="language-plaintext highlighter-rouge">inviteFriends</code> fully testable. Even after the full refactoring, the code for <code class="language-plaintext highlighter-rouge">lookupFriends</code> would not change.</p>
<p>Instead, our type alias for <code class="language-plaintext highlighter-rouge">Webapp</code> would expand to include new environmental effects:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">type</span> <span class="kt">WebappFX</span> <span class="o">=</span>
<span class="nc">Database</span> <span class="k">with</span> <span class="nc">Social</span> <span class="k">with</span> <span class="nc">Email</span>
<span class="k">type</span> <span class="kt">Webapp</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=</span> <span class="nc">ZIO</span><span class="o">[</span><span class="kt">WebappFX</span>, <span class="kt">Throwable</span>, <span class="kt">A</span><span class="o">]</span></code></pre></figure>
<p>Now all the methods in the <code class="language-plaintext highlighter-rouge">DB</code>, <code class="language-plaintext highlighter-rouge">Social</code>, and <code class="language-plaintext highlighter-rouge">Email</code> objects would simply delegate to their respective modules using <code class="language-plaintext highlighter-rouge">ZIO.accessM</code>.</p>
<p>Running our application would now look a little different:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">myRuntime</span> <span class="k">=</span>
<span class="nc">Runtime</span><span class="o">(</span>
<span class="k">new</span> <span class="nc">DatabaseLive</span>
<span class="k">with</span> <span class="nc">SocialLive</span>
<span class="k">with</span> <span class="nc">EmailLive</span><span class="o">,</span> <span class="nc">PlatformLive</span><span class="o">)</span>
<span class="o">...</span>
<span class="nv">myRuntime</span><span class="o">.</span><span class="py">unsafeRun</span><span class="o">(</span><span class="n">effect</span><span class="o">)</span></code></pre></figure>
<p>Finally, testing <code class="language-plaintext highlighter-rouge">lookupFriends</code> would be entirely fast, deterministic, and type-safe, without any dependencies on external systems, or use of any reflection.</p>
<h2 id="summary">Summary</h2>
<p>Environmental effects make it easy to test purely functional applications—significantly easier and with less ceremony than tagless-final, with full type inference, and without false promises.</p>
<p>Morever, with environmental effects, we can make even just a <em>single function</em> testable, by pushing that function into an environmental effect. This requires just a few small changes that can be done incrementally without major disruption to the application.</p>
<p>This ability lets us make incremental progress towards better application architecture. We don’t have to solve all the problems in our code base at once. We can focus on making our application a little better each day.</p>
<p>While this post focused on ZIO Environment, if you’re using <code class="language-plaintext highlighter-rouge">Future</code>, Monix, or Cats IO, you can still use this approach with a <code class="language-plaintext highlighter-rouge">ReaderT</code> monad transformer. With a monad transformer, you will lose type inference and some performance, but you will gain the other benefits in a more familiar package.</p>
<p>If you’d like to give ZIO Environment a try, hop over to the <a href="https://github.com/scalaz/scalaz-zio">Github project page</a>, and be sure to stop by the <a href="https://gitter.im/scalaz/scalaz-zio">Gitter channel</a> and say hello.</p>
<p>In future posts, I will cover how to provide partial dependencies, how to model services that require other services (the <em>graph problem</em>), how to hide implementation details, and how this approach differs from the classic cake pattern. Stay tuned for more!</p>
<p><a href="https://degoes.net/articles/testable-zio">Testing Incrementally with ZIO Environment</a> was originally published by John A De Goes at <a href="https://degoes.net">John A De Goes</a> on March 07, 2019.</p>