When Treasury Secretary Janet L. Yellen thinks about restoring the millions of jobs lost to the pandemic and making the economy more equitable, her mind sometimes turns to a place that shaped her views: Bay Ridge, Brooklyn.

The daughter of a doctor, Ms. Yellen, 74, grew up in the shadow of the Brooklyn waterfront and watched as longshoremen and laborers who were struggling with health problems and lost jobs came to her father’s office on the bottom floor of their family’s brownstone. Ms. Yellen, who would sit on her stoop and listen to their stories, became interested in how economics could help.

On Wednesday, Ms. Yellen recalled those days during a virtual keynote address to new graduates of Fort Hamilton High School, where she was valedictorian of the class of 1963. It was in her old neighborhood, she said, where she learned to measure success through the lens of public service.

“The lesson Bay Ridge taught us was that a happy and successful life — like a happy, successful community — depends not just on others’ love and respect for you, but, even more, on your love and respect for them,” Ms. Yellen told the students in a recorded speech that was streamed to their smartphones.

In a brief interview ahead of the graduation, Ms. Yellen explained how growing up in a working class enclave of Brooklyn was formative.

“One of the reasons I became an economist and ended up focusing on the labor market and on employment and things like that, is because of what we saw growing up, particularly with my father’s patients,” Ms. Yellen said. “The ups and downs in the economy, people would lose jobs and there were a lot of stories growing up of problems that families had.”

Back then, immigrants from Scandinavia, Ireland and Italy populated the neighborhood that is encircled by the Belt Parkway on the west and the Gowanus Expressway on the east. These days the demographics have shifted to include more Hispanics and immigrants from China, Russia, Greece and the Middle East.

The pandemic hit many of them hard.

Ms. Yellen explained that before the 1980s, fortunes tended to rise more or less with the broader economic tides. That has shifted in recent decades as structural problems like access to quality education, health care and child care helped perpetuate economic inequalities. That downside of that has been felt acutely in areas where immigrants and minorities live.

“Here you have a pandemic that has disproportionately fallen on low-income workers and minorities who are struggling anyhow in this economy, and knowing what unemployment is like, which is really what I’ve been thinking about, more or less my whole life,” Ms. Yellen said. “Of course it immediately focused us on wanting to help, intervening in ways that would enable people to get back on their feet, get the economy operating again and creating jobs as rapidly as possible.”

The sentiment comes at a pivotal moment in the early days of Ms. Yellen’s tenure as Treasury secretary. The Biden administration is engaged in talks with Republicans over the fate of its $4 trillion jobs and infrastructure proposals, Ms. Yellen is in the midst of fraught international negotiations over an overhaul of the global tax system and inflation is looming as a threat to the economic recovery.

Ms. Yellen served as Fed chair from 2014 to 2018 and has faced criticism for exacerbating some of the economic trends that she laments. Beginning in 2015, she oversaw a series of rate increases that were historically slow, but which many economists have come to see as premature. Higher rates may have taken some of the momentum out of the economy, making for a slower job market rebound and weaker-than-necessary wage gains, especially at the bottom of the income scale.

At the same time, some conservatives have said that the Fed’s low-rate policies exacerbate wealth inequality, a characterization former central bank officials including Ben S. Bernanke dispute.

After leaving the Fed in 2018, Ms. Yellen went on to earn millions of dollars making paid speeches to the kinds of financial firms that she says should pay more in taxes.

Ms. Yellen’s parents lived through the Great Depression, but her youth was comfortable, studded with academic accolades and intellectual hobbies, including a rock collection. She would take 5-cent ferry rides to Staten Island to play in Clove Lakes Park with friends. A trip to the American Museum of Natural History with her brother sparked an interest in gems. And in her junior year of high school, Ms. Yellen took a geology course on Saturdays and then would meet up with friends around New York City to participate in a rock collection club.

Although there has been speculation that Ms. Yellen is also an avid stamp collector, the Treasury secretary said such rumors are “hooey.” In fact, she says, she has been keeping the collection of her late-mother in a safe deposit box for many years.

Despite her early interest in economics, Ms. Yellen told the class of 2021 that her passion in high school was actually journalism. As a reporter for The Pilot, the Fort Hamilton High School newspaper, she covered the construction of the Verrazzano-Narrows Bridge and helped uncover Bay Ridge’s forgotten Revolutionary War cemetery. A trove of clippings from the newspaper unearthed by The New York Times in 2013 revealed that she once penned an absurdist column on current affairs called “Yellen Around.”

At the graduation ceremony, she told the students that journalism taught her to embrace assignments that at the time seemed small. The lesson was helpful when she started a new career at the Federal Reserve as a policymaker after being denied tenure at Harvard University — a blow to a someone who was named “class scholar” by the yearbook editors at Fort Hamilton High School.

“And my previous disappointment vanished from my mind,” Ms. Yellen said. “I had discovered a new joy.”

These days, Ms. Yellen has hero status at Fort Hamilton High School, which was named after a military installation that honors Alexander Hamilton, the first Treasury secretary.

“She is our most famous graduate,” said Kaye Houlihan, the school’s principal, adding that Ms. Yellen’s name often comes up in the school’s economics classes and as an example of where a Fort Hamilton education can lead.

Jeanna Smialekcontributed reporting.