Staff Engineer/Tech Lead For ML Feature Engineering

Twitter is what’s happening and what people are talking about right now. For us, life's not about a job, it's about purpose. We believe real change starts with conversation. Here, your voice matters. Come as you are and together we'll do what's right (not what's easy) to serve the public conversation.

Cortex empowers internal teams to efficiently leverage ML by providing a platform and by unifying, educating, and advancing the state of the art in ML technologies within Twitter.

We win when our customers win by helping our users stay informed, share and discuss what matters; by serving the public conversation. We’re building an AI-first company and every major initiative is increasingly dependent on the successful application of machine learning. Cortex is at the nexus of this evolution. Our team of ML software engineers is constructing one of the strongest machine learning platforms in the world, based on the latest ML industry practices, deep learning, engineering excellence, and powered by Twitter data at scale.

The ML Feature Management (MLFM) team is part of the Cortex Platform group. Together with the other Cortex Platform teams, we develop tools and infrastructure that standardizes machine learning development at Twitter on a common platform. This allows ML engineers in our customer teams to do their work better and faster and with a more modern stack that follows broad industry trends.

More specifically, our team’s vision is to maximize the velocity of Feature Engineering. To date, we’ve built and heavily invested in Twitter’s Feature Store, to share features across different ML product teams in production and thereby to broaden the impact of feature engineering investments across the company. This has been a wildly successful mission over the last two years and has prompted all ML teams to migrate to the Feature Store which is expected to complete in 2020.

The next frontier is to improve the tooling across the entire feature lifecycle. Some examples of tools we expect to explore along the way are: statistical methods for opportunity sizing feature ideas; Feature Store notebook integration; automatic estimation of capacity requirements and dollar cost of deploying a new feature; easy production testing of new features with A/B tests; tools for analyzing feature importance and removing features that have expired.

In the near term, we envision a series of prototypes, to be validated with pilot partners for a few quarters. Winning prototypes will then be developed into mature products. We believe this has the potential for a 10x productivity improvement of feature engineers within two years. In the longer term, this could lead to model-based semi-automated feature recommendations and data-driven decision-making tools for ML practitioners.

What you’ll do

We’re looking for a team member to join us in staff engineer or tech lead capacity to head up this effort, together with 2-4 engineers. Across the industry to date, feature engineering workflows remain fragmented and employ highly bespoke tools. Unifying and consolidating the toolchain along the feature lifecycle and thereby establishing standard methodologies for feature engineers and modelers we believe can deliver order-of-magnitude gains in productivity. If this sounds like you, come join us and turn this industry-leading opportunity into a company-wide competitive advantage!

Since this project is brand new and there are no industry peers to follow that have published end-to-end solutions, we expect a fair amount of uncertainty and ambiguity. You are not only comfortable with ambiguity but view it as an opening to quickly explore a multitude of options. You bring the knowledge and experience to build out validated ideas into full-fledged products for our ML customer teams. Keeping a portfolio of product ideas at different stages of maturity in flight and producing a steady cadence of robust product innovation is your primary M.O.

Who you are

Do you identify with the majority of the following traits? Yes? We believe they will make you successful in this role.

• You have a passion for machine learning and developer tools.

• You’re an innovator and entrepreneur with a track record of discovering “product-market fit” for seed-stage ideas and growing them into viable saplings.

• You bring partners together across organizational and functional boundaries.

• You’re able to articulate a clear vision and enroll the team and partners into it, both in spoken and written form, while remaining open to a constructive dialogue.

• You lead by example with technical strength and rigor in process (such as formulating and testing product hypotheses).

• You are sufficiently organized to keep an emerging product effort with lots of uncertainties on track.

• You multiply the effect of contributors by inspiring and growing them on and off the team across different levels of seniority, skills and geographical boundaries.

• You’re motivated by delivering impactful products that accelerate the feature engineering efforts of our customers.

• You’ve got a working knowledge of Jupyter notebooks and Python, plus experience with a compiled language, such as Scala or Java.

By nature of the problem domain, we expect you to have experience in:

• 7+ years in building and delivering working software through an iterative, agile process.

• 2+ years in Sr/Staff engineering capacity with demonstrated leadership skills. (This role does not include people management responsibilities.)

• 5+ years of work experience in software engineering in the areas of distributed data processing, developer tooling and/or ML platform space.

• 2+ years experience with ML problems and tools either through first-hand modeling or close collaboration with modeling engineers or data scientists.

• Entrepreneurial experience is a distinct plus.

• M.S. or Ph.D. degree in computer science or a related field or equivalent work experience.

All your information will be kept confidential according to EEO guidelines. We are committed to an inclusive and diverse Twitter. Twitter is an equal opportunity employer. We do not discriminate based on race, color, ethnicity, ancestry, national origin, religion, sex, gender, gender identity, gender expression, sexual orientation, age, disability, veteran status, genetic information, marital status or any legally protected status.

We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation.